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7th  Annual  CMMI  Technology  Conference  &  User  Group 

Investigation f  Measures  and  Lessons  Learned  about  the  Relationship  between  CMMI®  Process  Capability  and 

Project  or  Program  Performance^^ 

Denver,  Colorado 
November  12  -  15,  2007 


MONDAY,  NOVEMBER  12,  2007 


•  CMMI  VI  .2  —  An  Overview  Mr.  David  Phillips,  SEI 


TUESDAY,  NOVEMBER  13,  2007 


State  of  CMMI® 

•  Mr.  Clyde  Chittister,  Chief  Operating  Offieer,  SEI 
Exeeutive  Panel 

Panelists: 

Ms.  Kristen  Baldwin,  Offiee  of  the  Seeretary  of  Defense 
Mr.  Tom  Neff,  Defense  Threat  Reduetion  Ageney 
Mr.  Rieh  Frost,  General  Motors 

Luneh  with  Guest  Speaker 

•  Mr.  Mark  Sehaffer,  Direetor,  Systems  &  Software  Engineering,  OSD  (AT&L) 

Technical  Sessions 

TRACK  1 

•  When  the  Only  Tool  You  Have  is  a  Hammer,  Every  Problem  Begins  to  Look  Like  a  Nail,  Mr.  Sam  Fogle,  ACE  Guides,  EEC 

•  The  Journey  to  CMMI  Level ,  Mr.  Andrew  Lay,  Loekheed  Martin  Aeronauties  Company 

•  Visualizing  Improvement  with  Capability  Waypoints,  Mr.  Robert  Jaeob, 

Naval  Air  Systems  Commandinstitutionalization  Measures:  Key  to  Improved  Proeess  Monitoring,  Dr.  John  Rusnak,  Loekheed  Martin  Spaee  Systems 
Company 

TRACK  3 

•  Assuring  Quality  for  Effieient  &  Suffieient  Testing  Mr.  Pramod  Varma,  Wipro  Teehnologies 
TRACK  4 

•  Bridging  Proeess  Improvement  During  Program  Management  Evolution:  An  Experienee  Report  Capt  DeWitt  Latimer,  USAF 

•  An  “Embedded  SCAMPI-C”  Appraisal  at  the  National  Seeurity  Ageney.  Mr.  Joseph  Wiekless,  SEI 

TRACKS 

•  Linking  Projeet  Performanee  to  CMMI  Proeess  Capability  through  Lean  Measurements,  Mr.  Jeffrey  Dutton,  Jaeobs  Teehnology 

•  Quantitative  Models  for  Predieting  Projeet  Sueeess,  Dr.  Riek  Hefner,  Northrop  Grumman  Corporation 

TRACK  6 
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•  How  to  Kick  Start  a  Process  Improvement  Effort  to  Achieve  a  CMMI  Rating,  Ms.  Brenda  Hall,  Computer  Sciences  Corporation 

•  SEI  Appraisal  Program  Quality  Report,  Mr.  William  Hayes,  SEI 

•  The  Process  In-execution  Review  (PIER)  After  Three  Years,  Mr.  Dale  Swanson,  The  MITRE  Corporation 

•  I’m  Preparing  My  Organization  for  an  Appraisal,  but  I’m  Not  Really  Sure  I  Understand  this  PHD  Thing.  Should  I  Worry?,  Mr.  Sam  Fogle,  ACE  Guides,  EEC 
TRACK? 

•  Aligning  CMMI  and  ITIL  -  Where  Am  I  and  Which  Way  Should  I  Go,  Mr.  Pat  Mitryk,  Cognence,  Inc 

•  Integrated  System  Framework:  A  Way  Out  of  the  Multi-Model  Madness,  Mr.  Paul  Byrnes,  Integrated  System  Diagnostics 


WEDNESDAY,  NOVEMBER  14,  2007 


Lunch  with  Guest  Speaker 

•  Ms.  Mary  Poppendieck,  President,  Poppendieck,  EEC 
Technical  Sessions 

TRACK  1 

•  CMMI  Contenders,  CMMI  Pretenders,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Initial  Fears  of  CMMI  Introduction  and  How  Things  Really  Played  Out,  Dr.  Paul  Nugent,  General  Dynamics  Advanced 
Information  Systems 

•  Software  Firm  +  CMMI  Level  2  Initiative  +15  months  =  Dramatic  Quality  Improvements,  Mr.  Jeff  Simpson,  Campus  Management  Corporation 

•  How  to  Explain  the  Value  of  Every  CMMI  Practice,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Mrs.  Doubtfire  Answers  Your  Questions  about  Process  Improvement,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Developing  a  Second  Generation  Directive  System  Architecture,  Mr.  Kenneth  Weinberg,  Raytheon  Company 

•  Whose  Processes  Are  These,  Anyway,  Ms.  Judith  Tejan,  AAI  Services  Corporation 

•  Scientific  Breakthroughs  in  Process  Improvement,  Ms.  Cheryl  White,  Change  Delivery  Group 

TRACK  2 

•  The  What,  When,  Why  and  How  for  CMMI  Training,  Mr.  Tom  Bragg,  AVISTA  Incorporated 

•  Transitioning  to  the  CMMI:  What  They  Never  Told  You,  Mr.  Steve  Fried,  The  Boeing  Company 

•  CMMI  Implementation:  Overcoming  the  PPQA  Challenge,  Mr.  Pat  Mitryk,  Cognence,  Inc. 

•  How  to  Measurably  Improve  Your  Requirements,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Inc.  (QIC) 

TRACKS 

•  Using  Lean  Six  Sigma  to  Implement  CMMI  High  Maturity  Practices,  Ms.  Beth  Clark,  Lockheed  Martin 

•  The  Potential  for  Lean  Acquisition  of  Software  Intensive  Systems,  Mr.  Jeffrey  Dutton,  Jacobs  Technology 

•  Lean,  CMMI  and  Six  Sigma  Working  Together  to  Achieve  High  Success,  Ms.  Susan  Bassham,  US  Army  Aviation  and  Missle  Command 

•  Comparing  and  Contrasting  the  PP  &  PMC  Process  Areas  of  CMMI  v  1.2  and  SCRUM,  Dr.  Aldo  Dagnino  ABB,  Inc.  -  US  Corporate  Research 

•  Effective  Systems  Engineering:  What’s  the  Payoff  for  Program  Performance?,  NDIA  Systems  EngineeringsEffectiveness 

•  What’s  All  this  ‘churn’  in  Systems  Engineering  Standards  and  Models!?,  Mr.  Donald  Gantzer,  SAIC 

TRACK  4 

•  Driving  Process  Improvement  Using  the  CMMI-ACQ  at  General  Motors,  Dr.  Richard  Frost,  General  Motors 

•  Leading  Indicators  for  Acquisition  Programs,  Mr.  Robert  Ferguson,  SEI 

•  CMMI  High  Maturity  Misconceptions,  Mr.  William  Hayes,  SEI 

•  High  Maturity:  How  Do  We  Know?,  Dr.  Mike  Konrad,  SEI 

•  High  Maturity  System/Software  Cost  Estimation,  Dr.  Richard  Welch,  Northrop  Grumman  Corporation 

•  ADVANCE  -  Implementing  a  Defect  Model  for  Performance  Prediction,  Mr.  Stanley  Martin,  L-3  Communications/IS 

•  Statistically  Managing  a  Critical  Logistics  Schedule  Using  CMMI,  Mr.  Robert  Tuthill,  Northrop  Grumman  Corporation 

•  A  More  Practical  Set  of  High  Maturity  Practices,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

TRACK  5 

•  Program  Level  Return  on  Investment  for  CMMI®  Process  Improvement,  Mr.  J  Perry,  BAE  Systems 

•  How  Do  We  Get  on  the  Road  to  Maturity?,  Mrs.  Debra  Perry,  Harris  Corporation 

•  Understanding  CMMI  Measurement  Capabilities  Performance  &  Outcomes:  Results  from  the  2007  SEI  State  of  Measurement  Practices  Survey,  Dr.  Dennis 
Goldenson,  SEI 

•  Using  Predicted  Delivered  Defects  as  a  Management  Tool,  Mr.  Dustin  Sims,  BAE  Systems 

•  Calibrating  the  Project  Planning  Process,  Mr.  Donald  Corpron,  Northrop  Grumman  Corporation 

•  All  Others  Bring  Data,  Ms.  Charlene  Gross,  SEI 

TRACK  6 

•  Executing  a  Successful  CMMI  Maturity  Level  3  Scampi  for  Spawar  Systems  Center  Charleston,  Mr.  Michael  Kutch, 

SPAWAR  Systems  Center  Charleston 
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•  CMMI  SCAMPI  Appraisals  -  The  People/The  Process/The  Results-United  Space  Alliance,  LLC  Lessons  Learned,  Ms.  Robin  Hurst,  United  Space  Alliance, 
LLC 

•  Proposed  Approach  to  Heterogeneous  CMMI  Appraisals,  Mr.  Joseph  Vaudeville,  Northrop  Grumman  Corporation 

•  Selecting  a  Representative  Sample  for  CMMI  Enterprise  Appraisals,  Ms.  Kathryn  Kirby,  Raytheon  Company 

•  Logistics  and  Lessons  Learned  in  Conducting  an  CMMI®  Maturity  Level  3  Full-Model  Scope  Enteprise-level  Appraisal 
Ms.  Kathryn  Kirby,  Raytheon  Company 

TRACK? 

•  Excellence  at  the  Organization,  Team  and  Individual  levels;  CMMI,  TSP  and  PSP  -  Experience,  Lessons  Learned  and  Why  all  Three  are  Needed,  Mr.  Girish 
Seshagiri,  Advanced  Information  Services,  Inc. 

•  IEEE  Fife  Cycle  Standards  and  the  CMMI®  -  Implementation  Considerations,  Dr.  Peter  Hantos,  The  Aerospace  Corporation 

•  Using  CMMI  and  OPM3  to  Improve  Performance,  Mr.  Thomas  Keuten,  Pariveda  Solutions 

•  Complementary  or  Competing?  Achieving  Synergy  with  OPM3®,  CMMI®,  and  ISO  9001-2000,  Mr.  Mark  Scott,  Harris  Corporation 

•  Formal  Process  Definition  with  Industry  Standards,  Mr.  Chris  Armstrong,  Armstrong  Process  Group,  Inc. 

•  Project  Management  Architecture  Design  as  a  Critical  Success  Factor  in  CMMI  Model  Implementation,  Mr.  Christen  MacMillan,  E-3  Communications 


THURSDAY,  NOVEMBER  15,  2007 


Lunch  and  Award  Presentation 
TRACK  1 

•  Fast  Track  to  Higher  CMMI  Maturity  Levels:  Lessons  Learned  from  Five  Initiatives,  Ms.  Cheryl  White,  Change  Delivery 

•  Seven  Success  Factors  for  CMMI  Based  Process  Improvement,  Mr.  Orhan  Kalayci,  XPI  -  extreme  Process  Improvement 

•  CMMI  Process  Improvement:  It’s  Not  a  Technical  Problem,  It’s  a  People  Problem!,  Mr.  Rolf  Reitzig,  Cognence 

•  Improving  Project  Proposal  Quality  via  CMMI,  Mr.  Chen  Wang,  Institute  for  Information  Industry 

TRACK  2 

•  A  Framework  to  Manage  and  Evaluate  Remote  Software  Testing  Using  CMMI,  Dr.  Aldo  Dagnino,  ABB,  Inc.  -  US  Corporate  Research 

•  CMMI,  Configuration  Management,  and  Baseball  -  How  to  Score,  Ms.  Julie  Schmarje,  Raytheon  Company 

•  Automated  Systems  for  Project  Portfolio  Management  -  Project  Success  and  Outstanding  Earned  Value,  Mr.  Pothiraj  Selvaraj, 

Global  Computer  Enterprises 

TRACK  3 

•  Project  Management  by  Functional  Capability,  Mr.  Fred  Schenker,  SEI 

•  Software  Architecture  Development  Eeveraging  the  Attribute  Driven  Design  and  CMMI  Methodologies,  Dr.  Aldo  Dagnino, 

ABB,  Inc.  US  Corporate  Research 

•  Systems  Assurance  -  Practices  Make  Perfect  -  How  Your  Engineering  and  Management  Practices  Can  Help  Meet  the  Assurance  Challenge,  Mr.  Paul  Croll, 
Computer  Sciences  Corporation 

•  Tools  and  Resources  to  Enable  Systems  Engineering  Improvement,  Mr.  Michael  Kutch,  SPAWAR  Systems  Center  Charleston 

•  Applying  CMMI  Principles  to  Certification  Process  of  Legacy  Aircraft,  Ms.  Michele  Bruno,  The  Boeing  Company 

•  Accreditation  of  Undergraduate  Programs  in  Computing,  Software  Engineering,  Systems  Engineering  and  the  Ties  to  CMMI-based  Improvement,  Mr.  Dan 
Nash,  Raytheon  Company 

•  How  Future  Trends  in  Systems  and  Software  Engineering  Bode  Well  for  Enabling  the  Rapid  Adoption  of  CMMI,  Dr.  Ken  Nidiffer,  SEI 
TRACK  4 

•  Thought  Before  Action:  A  High  Maturity  Roadmap  for  the  Lower  Maturity  Organization,  Mr.  James  McHale,  SEI 

•  Integrated  Implementation  of  Advanced  Maturity  Practices,  Mr.  Dale  Childs,  DFAS 

•  Process  Performance  Baselines  and  Models:  Duh,  I  Don’t  Get  It,  Ms.  Diane  Mizukami- Williams,  Northrop  Grumman  Mission  Systems 

•  Expanding  Statistical  Process  Control  Across  All  Engineering  Disciplines:  A  Sequence  of  Practical  Case  Studies,  Dr.  Richard  Welch,  Northrop  Grumman 
Corporation 

•  Statistical  Process  Control  Applied  to  Specification  Requirements  Process,  Mr.  A1  Florence,  The  MITRE  Corporation 

•  Implementing  High  Maturity  in  a  Production  Support  Environment,  Ms.  Virginia  Slavin,  SSCI 

•  Using  the  Scientific  Method  at  Eevels  4-5,  Dr.  Jeff  Ricketts,  Raytheon  Company 

TRACK  5 

•  The  Productivity  Puzzle,  Mrs.  Jill  Brooks,  Raytheon  Company 

•  Using  Metrics  to  Develop  a  Software  Project  Strategy,  Mr.  Donald  Beckett,  Quantitative  Software  Management 

•  Eessons  Teamed  in  the  Implementation  of  Measurement  Techniques  for  CMMI  GP  2.8,  Dr.  Susanna  Schwab,  L-3  Communications 

•  Optimizing  the  Measurement  Process,  Mr.  Gary  Natwick,  Harris  Corporation 

•  Measurement  Strategies  in  the  CMMI,  Dr.  Rick  Hefner,  Northrop  Gmmman  Corporation 

•  5  Major  Sites,  4  Separate  Disciplines,  1 1,500  Engineers,  1  Data  Repository:  Having  Data  You  Can  Actually  Use  -  Priceless! 

Mrs.  Jill  Brooks,  Raytheon  Company 

TRACK  6 
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•  Cutting  Appraisal  Costs  in  Half,  Dr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Experiences  Implementing  Very  Large  High  Confidenee  Enterprise  Appraisals,  Mr.  Paul  Byrnes,  Integrated  System  Diagnosties 

•  Proeess  Complianee  the  Smart  Way,  Mr.  Gary  Natwiek,  Harris  Corporation 

•  Judging  the  Suitability  of  Alternative  Praetiees,  Dr.  Riek  Hefner,  Northrop  Grumman  Corporation 

•  Lessons  Learned  Condueting  High  Maturity  SCAMPIs,  Mr.  Paul  Byrnes,  Integrated  System  Diagnosties 

•  Benefits  of  SCAMPI  Class  C  in  Small  Settings,  Dr.  Mary  Anne  Herndon,  Transdyne  Corporation 

•  Lower  Cost,  More  Effeetive  Alternatives  to  SCAMPIs,  Dr.  Riek  Hefner,  Northrop  Grumman  Corporation 

•  Using  Workshops  to  Speed  CMMI  Adoption  and  Evidenee  Gathering,  Dr.  Riek  Hefner,  Northrop  Grumman  Corporation 

TRACK? 

•  Quality  Maturity  Model  -  Foundation  for  Proeess  Institutionalization,  Mr.  Sumit  Gupta,  Royal  Bank  of  Seotland  -  India  Development  Center 

•  Not  Just  for  Software  Anymore:  Lessons  Learned  From  a  CMMI™  Appraisal  on  Projeets  in  a  Nonnuelear  Weapons  Faeility, 

Mr.  Daniel  Fritts,  Honeywell 

•  CMMI  for  Serviees  Overview,  Mr.  Craig  Hollenbaeh,  Northrop  Grumman  Corporation 

•  Defining  Lean  Serviee  and  Maintenanee  Proeesses  that  are  CMMI  Compliant,  Mr.  Timothy  Olson,  Lean  Solutions  Institute,  Ine.  (QIC) 

•  Implementing  Aequisition  and  System  Engineering  Proeesses  in  a  Maintenanee  Organization,  Mr.  Bill  Feteeh,  The  MITRE  Corporation 
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CMMI  is  registered  in  the  US  Patent  &  Trademark  Office  by  Carnegie  Meiion  University 


Conference  Agenda 


SUNDAY,  NOVEMBER  11,  2007 


3:00  PM  -  6:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

MONDAY,  NOVEMBER  12,  2007 

The  Tutorial  sessions  require  a  $275  registration  fee  which  is  in  addition  to  the  Conference  registration  fee. 

7:00  AM -7:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:00  AM -8:00  AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:00  AM-  5:30  PM 

Tutorial  Sessions  (must  be  registered) 

Refer  to  Following  Page 

9:45  AM-  10:15  AM 

Break  (Tutorial  Attendees  Only) 

Grand  Mesa  Foyer 

12:00  PM  - 1:00  PM 

Lunch  (Tutorial  Attendees  Only) 

Grand  Mesa  ABC  Corridor 

2:45  PM -3:15  PM 

Break  (Tutorial  Attendees  Only) 

Grand  Mesa  Foyer 

5:30  PM  -  7:00  PM 

Reception  (Open  to  all  Attendees) 

Atrium  Display  Area 

TUESDAY,  NOVEMBER  13,  2007 

7:15  AM -7:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:15AM-8:15AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:15  AM -8:30  AM 

Welcome  &  Opening  Remarks 

Grand  Mesa  DEF 

♦  Mr.  Sam  Campagna,  Director,  Operations,  NDIA 

♦  Mr  Bob  Rassa,  Director,  Systems  Support,  Raytheon 

Company 

8:30  AM -9:15AM 

State  of  CM  Ml® 

Grand  Mesa  DEF 

♦  Mr  Bob  Rassa,  Director,  Systems  Support,  Raytheon 

Company 

9:15  AM-  10:00  AM 

♦  Mr  Clyde  Chittister,  Chief  Operating  Officer  SEI 

CM  Ml®  Into  the  Future 

Grand  Mesa  DEF 

♦  Mr  Bob  Rassa,  Director,  Systems  Support,  Raytheon 

Company 

10:00  AM  -  10:15  AM 

Break 

Grand  Mesa  Foyer 

10:15  AM  -  11:45  AM 

Executive  Panel 

Grand  Mesa  DEF 

12:00  PM  -  1:30  PM 

Moderator: 

Mr  Bob  Rassa,  Raytheon  Company 

Panelists: 

Ms.  Kristen  Baldwin,  Office  of  the  Secretary  of  Defense 
Mr  Tom  Neff,  Defense  Threat  Reduction  Agency 

Mr  Rich  Frost.  General  Motors 

Mr.  Mike  Phillips,  Software  Engineering  Institute 

Lunch  with  Guest  Speaker 

Grand  Mesa  ABC  Corridor 

♦  Mr.  Mark  Schaffer,  Director  Systems  &  Software  Engineering,  OSD  (AT&U 

1:30  PM -5:00  PM 

Technical  Sessions 

Refer  to  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Grand  Mesa  Foyer 

5:00  PM  -  6:30  PM 

CMMI-ACQ  Rollout  Reception 

Atrium  Display  Area 

WEDNESDAY,  NOVEMBER  14,  2007 

7:15  AM -5:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:15AM-8:15AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:15  AM-  11:45  AM 

Technical  Sessions 

Refer  to  Following  Pages 

9:45  AM-  10:15  AM 

Break 

Grand  Mesa  Foyer 

12:00  PM  -  1:30  PM 

Lunch  with  Guest  Speaker 

Grand  Mesa  ABC  Corridor 

1:30  PM -5:00  PM 

♦  Ms.  Marv  PoDoendieck.  President.  Poooendieck.  LLC 
Technical  Sessions 

Refer  to  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Grand  Mesa  Foyer 

THURSDAY,  NOVEMBER  15,  2007 

7:15  AM -5:00  PM 

Conference  Registration  Open 

Grand  Mesa  Foyer 

7:15AM-8:15AM 

Continental  Breakfast 

Grand  Mesa  Foyer 

8:15  AM-  11:45  AM 

Technical  Sessions 

Refer  to  Following  Pages 

9:45  AM-  10:15  AM 

Break 

Grand  Mesa  Foyer 

12:00  PM  -  1:30  PM 

Lunch  and  Award  Presentation 

Grand  Mesa  ABC  Corridor 

1:30  PM -5:00  PM 

Technical  Sessions 

Refer  to  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Grand  Mesa  Foyer 

RECEPTION  (5:30  PM  -  7:00  PM)  (ALL  ATTENDEES) 
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Developer: 


How  big  is  this  project? 

I  doncbknow.  This  looks  really  hard. 

Well  we  need  to  know  how  big  it  is  so  we  can  estimate 
the  work. 

I4l  have  to  figure  out  how  hard  it  is  so  I  can  tell  you  how 
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The  project  manager  is  concerned  with  staffing  and  pianning  to  meet 
the  project®  objectives. 

The  project  manager  may  not  understand  what  the  engineer  means  by 
complexity. 

A  He  may  interpret  the  behavior  as  complaining. 

A  He  may  think  fHe  always  says  that,  but  it  doesnd  help  his  estimate.6 

The  project  manager  does  not  know  what  questions  to  ask, 
nor  has  he  thought  sufficiently  about  engaging  the  SE  in 
project  planning. 
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What  do  we  mean  by  the  word  ncomplexityd? 

What  methods  can  help  project  managers  resolve  complexity? 
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How  should  the  project  manager  question  the  staff  to  identify  the 
complexity? 
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The  project  manager  and  engineer  can  deal  with  the  complexity 
problem,  provided  that  each  understands  and  accepts  the  other® 
concern. 

A  The  project  manager  asks  the  right  kind  of  question. 

A  The  project  manager  is  amenable  to  creating  a  plan  that  will  allow  for 
resolution  of  the  complexity  by  the  engineering  staff. 

A  The  engineer  understands  how  the  project  plan  might  help  to  mitigate 
the  schedule  and  cost  problems  that  result  from  complexity. 

A  The  budget  and  schedule  are  not  so  tightly  constrained  that  the  project 
cannot  be  accomplished. 

The  remainder  of  this  talk  will  describe  some  planning 
actions  to  help  resolve  select  types  of  project  complexity. 
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Research 


Research  considering  the  complexity  in  product  development  projects 
A  Business  Schools 

6  Steven  Eppinger  and  Nelson  Repenning  at  MIT 

6  Kim  Clark  at  Harvard 

A  Engineering  School  papers 

6  AN  A.  Yassine  at  Univ.  Illinois  Urbana-Champagne 

Research  has  considered  task  structure  in  response  to  complex 
problems  in  product  development .. 
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Definitions  seem  to  relate  to  the  difficulty  in  learning  a  capability  that  a 
team  or  individual  does  not  currently  possess. 

A  fMcCabe  complexityo  indicates  difficulty  in  learning  to  maintain  a  set  of 
code. 

A  nTechnology  introductionoentails  learning  a  lot  of  different  things: 
design,  testing,  technical  communications,  manufacturing,  e 

A  Invention  is  discovering  (learning)  a  new  design  pattern 

Resolving  the  complexity  depends  on  some  learning  process  i 

A  The  organization  must  develop  new  capabilities. 

A  Some  iteration  or  experiment  is  required  for  a  satisfactory  solution. 

A  The  team  must  learn  to  work  together. 
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Different  learning  requirements  suggest  an  approach. 

Big 

The  work  has  to  be  divided  into  teams  or  sub-projects  in  order  to 
produce  a  result  soon  enough  that  it  has  value. 

Deep 

An  unfamiliar  design  pattern  is  required.  It  may  even  require  a  new 
invention. 

Conflicting  Goals  (Design  Tradeoff) 

Problem  requires  some  form  of  experimentation,  prototyping  or  other 
trade-off  analysis.  An  optimal  (but  not  perfect)  solution  is  expected. 
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Large  projects  require  multiple  work  groups  operating  simultaneously 
and  somewhat  independently. 

Potential  Problems 

A  Synchronizing  the  work  is  very  difficult.  Teams  must  sometimes  start 
work  on  incomplete  information. 

A  Individuals  who  fail  to  fully  participate  in  the  work  of  integrated  product 
teams  (IPTs)  place  additional  burdens  on  the  other  teams. 

Things  to  be  learned: 

A  Team  boundaries  (fWe  do  this.  You  do  that.  Here®  how  we  decide. c) 

A  How  to  handle  incomplete  information 
A  How  to  declare  completeness 
A  How  to  verify  and  validate  each  other®  work 
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Needs  a  picture  of  team  arid  wbs  structure 
Relationships  show  learning 


Structuring  the  teams 

A  Balance  the  workload  to  achieve  desired  schedule 
A  Teams  have  needed  skills  and  resources 
Product  Concerns 

A  Sufficiently  many  integration  points  to  demonstrate  learning  and 
product  progress  (depends  on  system  architecture) 

Required  activities 

A  Learning  to  work  together  (say  8-24  hours  face-to-face  time) 

A  Specific  understanding  of  interfaces  and  boundaries 

A  Describing  exactly  what  is  incomplete  and  how  the  act  of  completing 
may  affect  current  results. 
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Consider  that  rChange  Requestsoare  an  out-of-cycle  development 
request. 

A  i.e.  some  design  work  is  already  completed  and  now  has  to  be  re-done. 
Considerations 

A  Affected  work  products 
A  Affected  teams 
A  Coordination  aspects 
A  Ripple  effects 
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IPT 

A  Participation  and  battle  rhythm 
A  Convergence  on  interfaces 
A  Issues  and  rework  on  interfaces 
A  Decision  bottlenecks 

A  Design  structure  matrix  to  show  distance  between  team  members 
Architecture 

A  Design  structure  matrix  to  show  interdependency 
A  Structure  for  integration/verification 
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An  aspect  of  the  design  is  new  to  the  development  team. 

Potential  problems 

A  Capability  to  perform  may  be  missing  or  have  limited  capacity  for  work. 

A  Productivity  suffers  and  team  generates  a  lot  of  rework. 

A  Lack  of  progress  affects  other  teams  and  causes  synchronization 
problems. 

Things  to  be  learned 

A  What  technology  works  (algorithm,  material,  equipment,  technique)? 

A  How  and  when  does  it  work? 

A  How  do  we  utilize  it  in  the  current  product  development  project? 

A  Do  we  want  to  develop  capability  and  capacity  or  buy  it? 
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The  first  use  of  a  genetic  algorithm  in  the  application. 

A  Who  must  understand  the  mathematics? 

A  How  long  does  convergence  take? 

A  How  can  we  test  the  convergence  and  result? 

A  What  do  we  need  to  document  for  maintenance? 

A  What  unique  bugs  could  occur  in  this  type  application? 

A  How  will  this  technology  affect  manufacturing  and  setup? 
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Deep  problems  take  time,  but  not  many  people. 

A  Some  very  highly  skilled  individuals  will  not  be  available  to  the  larger 
team  for  while  the  deep  problem  is  addressed. 

A  If  these  people  multi-task,  the  time  required  will  be  much  longer. 
Required  activities 

A  A  deep  problem  is  not  hsolvedountil  the  organization  can  utilize  the 
technology  to  produce  the  final  product. 

A  Technology  transfer  tools,  events  and  mentoring 
Costs  and  Risk  Mitigation  require  investigating  alternative  solutions. 

A  Alternative  implementations  may  be  needed  in  the  interim,  but  may  not 
fully  meet  quality  attribute  objectives. 

A  Buy  required  technology  and/or  development  capacity  (risk  transfer) 
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Some  stakeholder  values  are  in  apparent  conflict. 

A  More  power  and  less  fuel  consumption 
A  Faster  performance  and  more  security 
A  Flexibility  to  install  devices  and  information  assurance 
A  Faster  product  delivery  and  more  robust  design 
Conflict  may  be  between  stakeholders  increasing  the  difficulty 
A  Theory  of  Constraints  work  may  help  with  conflict  resolution 
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Potential  problems 

A  Separate  teams  may  attempt  to  achieve  the  goals  independently.  Each 
team  then  changes  the  resulting  system  behavior  in  some  way  opposite 
the  other®  goal. 

A  Slow  decision  process 

A  Usually  requires  multiple  iterations  for  resolution. 

A  Conflict  not  exposed  soon  enough  for  appropriate  resolution. 

Things  to  be  learned 

A  What  are  the  important  interactions?  What  values  work? 

A  What  are  the  sensitivity  points  and  trade-offs  inherent  in  our  design 
(architecture)? 

A  How  can  we  see  that  our  required  iterations  are  converging? 
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These  problems  always  require  some  form  of  experimentation. 

A  Experiments  include  simulation,  scenario  analysis,  trade  studies  and 
prototype  products 

A  There  is  a  cost  to  experimentation  that  can  be  hard  to  plan. 

Required  Activities 

A  Identify  sensitivity  points  and  trade-offs. 

A  Check  modularity  against  team  structure  so  that  decision  involves  as 
few  teams  as  possible. 

A  Plan  some  number  of  iterations  before  capability  is  required. 

A  Create  extra  integration  points  to  show  that  complexity  was  actually 
resolved. 

A  Consider  transforming  problem  into  a  ftleepoproblem.  (Find  a 
technological  approach). 


Software  Engineering  Institute  CarnegieMellon 


SEI  Presentation  (Basic) 
Author,  Date 

©2006  Carnegie  Mellon  University 


20 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


A  conversation 

Defining  complexity  and  its  effects  on  projects 
Research  into  toois  and  methods 


-  Software  Engineering  Institute 


CamegieMellon 


SEI  Presentation  (Basic) 
Author,  Date 

©2006  Carnegie  Mellon  University 


21 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


ods 


Design  Structure  Matrix  (DSM) 

A  DSM  has  proved  to  be  a  fairly  successful  approach  to  partitioning  and 
analyzing  very  large  systems,  (picture) 


Three  Configurations  that  Characterize  a  System 


Relationship 


Parallel 


Sequential 


Coupled 


Graph 

Representation 


<3- 


Three  Configurations 
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Representation 
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DSM  Data 
Types 

Representation 

Application 

Analysis  Method 

Paititionins,  Tearins, 

Banding,  Snnulation  and 
Eigenvalue  Analysis 

Task-based 

Task'' Activity 
input/outpnt  relationships 

Project  scheduling,  activity 
sequencing,  cycle  time  reduction 

Paititioning,  Tearing, 

Banding,  Snnulation  and 
Eigenvalue  .:Analysis 

Parameter- 

based 

Parameter  decision  points 
and  necessary  precedents 

Low  level  activity  sequencing 
and  process  constmction 

Team-based 

Multi-team  interface 

characteristics 

Organizational  design,  interface 
management,  team  integration 

Clustering 

Component- 

based 

Mnlti-component 

reiationsliips 

System  architecting,  engineering 
and  design 

Clustering 
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This  matrix  represents  one  with  a 
lot  of  complexity. 

Modularity  and  team  arrangement 
is  not  clear. 

By  re-ordering  the  matrix  we  can 
achieve  a  better  team  structure 
and  better  modularity  of  both  task 
and  design. 
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Scheduling 

A  DSM  provides  useful  information  about 

6  Team  interdependencies  i  requires  exchange  of  incomplete 
knowledge  and  active  participation 

6  Component  interaction  T  Requires  documentation  and  tests 

6  Iteration  T  requires  planned  extra  steps 

Team  Learning 

A  Joint  scenario  work 

A  Simulations  of  work  flow 

A  Joint  inspections 

A  Facilitators  for  planning 
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Best  known  method  TRIZ  (treez) 

A  Addresses  both  fDeepoand  fConflicting  Goalso 

A  Consists  of  40  strategies  for  innovation  and  problem  solving. 

A  Applies  mostly  to  hardware  engineering 

6  Such  as  physical  separation  of  function 

6  Time-dependent  separation  of  function 

QFD  relates  design  goals  to  design  with  cost  elements  and  exposes 
conflicting  goals 

QAW,  AT AM  expose  many  conflicting  goals  problems 
Design  Structure  Matrix 

A  Has  potential  for  mathematical  approaches  such  as  fwork-eigenvectoro 
and  simulation  of  task  structure. 
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Dedicated,  highly  skilled  resources 

Knowledge  transfer,  process  implementation 

A  The  new  technology  has  to  be  adapted  to  the  rest  of  the  product 
development  team.  It  may  require  additional  resources. 

Validation  of  utility  of  results  (testing,  learning,  etc.) 

A  New  capability  will  include  design  patterns,  test  patterns, 
documentation  skills,  customer  support  skills,  etc. 

Highly  skilled  resources  are  not  always  good  at  technology  transfer. 
Senior  engineering  management,  developers  and  testers  all  need  to 
learn  something  from  a  deep  problem.  Some  participation  in  progress 
reviews  and  experiments  needs  the  support  of  these  other  people. 
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Alternative  method 

A  Parallel  teams  attempt  different  solutions 

A  Purchase  products  or  the  development  capacity  from  outside 
Experiments 

A  Trade  studies,  prototypes,  simulation 
Project  management  consideration 

A  Resolution  of  deep  problems  has  to  start  as  early  as  possible  or  the 
schedule  will  grow  while  capability  and  capacity  problems  are  resolved. 

A  All  methods  associated  with  deep  problems  have  the  possibility  of 
taking  a  very  long  time  to  resolve. 

A  It  is  essential  to  have  a  reasonable  method  at  the  time  of  integration 
even  if  the  solution  is  not  optimal. 
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Partitioning  of  rBigocan  aggravate  rConflicting-Goals.6 

A  Separation  of  concerns  approach  may  allow  engineers  to  view  their 
responsibility  for  <quality-attribute-A>  as  independent  from  <quality- 
attribute-B>  resulting  in  a  sub-optimal  design. 

Sonnetinnes  work  on  fDeepoproblems  results  in  fBigoor 
fConflicting-Goalsoproblems. 

A  As  when  the  primary  solution  to  the  Deep  problem  is  to  partition  it  into 
several  other  problems. 

Sonne  fConflicting-Goalsoproblems  can  be  addressed  algorithmically 
resulting  in  a  fDeepoproblem. 
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IPPD  goals  address  the  Big  problem  and  Conflicting  Goals  problem 

A  IPT  structure  is  key 

A  Must  monitor  IPT  learning  and  non-learning  (issues,  etc.) 

A  IPT  must  discuss  content  as  well  as  schedule  if  members  are  to  learn. 

A  Integrated  Product  concept  has  to  be  at  the  forefront  of  the  project 
manager®  attention  as  the  primary  near-term  goal  for  each  IPT. 

Technical  Solution 

A  Does  not  satisfactorily  address  Deep  problems 

A  We  must  include  specific  efforts  to  develop  the  competencies  and 
capabilities  of  staff  and  process  to  introduce  a  technical  innovation. 

A  Even  choosing  an  outside  supplier  for  the  solution  requires 
development  of  new  internal  capabilities. 
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Maturity  Level 

Process  Areas 

Optimizing 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

Quantitatively 

Managed 

Quantitative  Project  Management 

Organizational  Process  Performance 

Defined 

Organizational  Process  Focus 

Organizational  Process  Definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Acquisition  Technicai  Management 

Acquisition  Verification 

Acquisition  Vaiidation 

Decision  Analysis  and  Resolution 

Managed 

Acquisition  Requirements  Deveiopment 

Agreement  Management 

Project  Planning 

Project  Monitoring  and  Control 

Requirements  Management 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Solicitation  and  Supplier  Agreement  Deveiopment 
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We  can  teach  project  managers  and  systems  engineers  (architects)  to 
talk  with  each  other  about  compiex  probiems. 

This  taik  described  complexity  as  3  different  type  probiems. 

A  Big,  Deep,  Conflicting  Goals 

Addressing  each  type  of  complexity  caiis  for  different  project 
management  strategies. 

Each  strategy  must  address  the  technicai  problem,  product  integration, 
iearning  events  and  the  project  sociai  network. 

A  We  need  to  identify  ways  to  monitor  that  the  development  team  is 
actually  learning  as  a  means  of  checking  progress. 
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Process  —  a  sequence  of  steps  performed  for  a 
given  purpose  {IEEE) 


Software  process  —  a  set  of  activities,  methods, 
practices,  and  transformations  that  people  use  to 
develop  and  maintain  software  and  the 
associated  products  (SEI) 
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Recognized  Adoption  Issues 


“70%  of  tools  purchased  by  the  organizations  in  the 
surveys  are  never  used,  other  than  perhaps  in  initial 
trial 

25%  are  used  by  only  one  team  or  person  within 
each  organization 

5%  are  widely  used,  but  not  to  capacity.  Perhaps 
only  10%  of  the  capacity  of  the  tool  is  used.” 


From  Jerry  Weinberg’s  informal  tool  survey,  cited  in  Quality 
Software  Management  vol  4:  Anticipating  Change.  Dorset 
House,  1997. 
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November  15**’,  2007 
Rolf  W.  Reitzig 
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Run  operations  as  if  they  were  a  franchise 


and  Expanded  Features 


-  Every  business  process  is  standardized 

-  Employees  can  easily  be  successful  by  following  the 
processes  as  outlined 

-  Everyone  knows  how  to  perform  their  job 

-  Tasks  are  performed  similarly  on  a  repeatable  basis  and 
improved  based  on  experience 

•  A  quality  process  will  yield  a  quality  product 
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ig  Concepts 


•  Great  businesses  are  not  built  by  extraordinary  people, 
but  by  ordinary  people  doing  extraordinary  things 


•  To  achieve  this,  a  system  is  absolutely  essential  -  it 
becomes  the  tools  people  use  to  increase  productivity, 
to  get  the  job  done  in  a  way  that  differentiates 

•  If  you  haven’t  orchestrated  your  business,  you  don’t 
own  it! 


Source:  The  e-Myth  Revisited,  Michael  E.  Gerber,  HarperCollins  Publishers,  1995 
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Role 


s  management’s  job  to  develop  systems  and  tools  and 
teach  people  how  to  use  them 

Its  the  people’s  job  to  use  the  tools  and  to  recommend 
improvements  based  on  their  experience  with  them 


•  There  is  no  such  thing  as  undesirable  work,  only  people 
who  view  certain  kinds  of  work  as  undesirable  -  create 
an  environment  in  which  doing  certain  things  is  more 
important  than  not  doing  them 

•  Management  makes  sure  employees  understand  the 
idea  behind  the  work  they  are  being  asked  to  do 

•  Avoid  “Management  by  Abdication”! 

Source:  The  e-Myth  Revisited,  Michael  E.  Gerber,  HarperCollins  Publishers,  1995 
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Integrated  Project  Management 
Project  Monitoring  and  Control 
Risk  Management 
Supplier  Agreement  Management 
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Configuration  Management 
Measurement  and  Analysis 
Process  and  Product  Quality  Assurance 
Decision  Analysis  and  Resolution 
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Organizations  typically  invest  2%-4%  of  their  IT  budget 
on  engineering  improvement 


•  Organizations  engaged  in  an  engineering  improvement 
effort  experience  50%+  gains  in  productivity  and  a 
25%+  decreases  in  post-release  defects 


•  Average  ROI  was  5:1 

•  Example:  An  IT  department  with  a  $100M  budget 
spending  $4M  on  SPI  can  expect  a  $20M  gain  in 
productivity  over  2  years 
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inciples  of  SPI 


1 .  Major  changes  to  the  software  process  must  start  at 
the  top 


2.  Effective  change  requires  a  goal  and  knowledge  of  the 
current  process 


3.  Software  process  improvement  requires  investment 

4.  Ultimately,  everyone  must  be  involved 

5.  Software  process  changes  will  not  be  retained  without 
conscious  effort  and  periodic  reinforcement 


6.  Change  is  continuous 

Source:  Humphrey,  W.S.  Managing  the  Software  Process.  Addison-Wesley,  1989 
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cepts 


1 .  To  improve  the  software  process,  someone  must  work 
on  it 

2.  Unplanned  process  improvement  is  wishful  thinking 

3.  Automation  of  a  poorly  defined  process  will  produce 
poorly  defined  results 

4.  Improvements  should  be  made  in  small,  tested  steps 

5.  Train,  train,  train! 


Source:  Humphrey,  W.S.  Managing  the  Software  Process.  Addison-Wesley,  1989 
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T  ransformation 


•  Improvement  models  like  CMMI  build  on  organizational 
transformation  theory  to  drive  effectiveness. 


•  Thus,  it  is  imperative  to  understand  organizational 
transformation  theory  in  order  to  implement  a 
franchisable  engineering  system  and  improve  results. 
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s  T ransformation  Best  Practices 


1 .  Establish  a  sense  of  urgency 

2.  Create  the  guiding  coalition 

3.  Develop  a  vision  and  strategy 

4.  Communicate  the  change  vision 

5.  Empower  employees  for  broad-based  action 

6.  Generate  short-term  wins 

7.  Consolidate  gains  and  produce  more  change 

8.  Anchor  new  approaches  in  the  culture 


Source:  John  P.  Kotler,  Leading  Change,  Harvard  Business  School  Press,  1996 
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g  a  Sense  of  Urgency 


•  Progression  to  subsequent  organizational 
transformation  phases  is  difficult,  if  not  impossible, 
unless  most  managers  honestly  believe  that  the  status 
quo  is  unacceptable 
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Guiding  Coalition 


•  Successful  transformations  must  be  guided  by  a 
powerful  coalition  that  can  act  as  a  team 


•  The  coalition  is  needed  because  no  one  individual  has 
the  information  needed  to  make  all  major  decisions  or 
the  time  and  credibility  needed  to  convince  lots  of 
people  to  implement  the  decisions 
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a  Vision  and  Strategy 


•  Vision  refers  to  a  picture  of  the  future  with  some  implicit 
or  explicit  commentary  on  why  people  should  strive  to 
create  that  future. 

•  3  purposes 

-  Clarifies  the  general  direction  for  change 

-  Motivates  people  to  take  action 

-  Coordinates  the  efforts  of  different  people 


•  Must  be  conveyable  in  5  minutes  or  less 
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ating  the  Change  Vision 


The  real  power  of  a  vision  is  unleashed  when  most  of 
those  involved  in  an  enterprise  have  a  common 
understanding  of  its  goals  and  direction 

You  cannot  overcommunicate  the  vision! 


•  A  common  mistake  by  the  guiding  coalition  is  to  assume 
the  organization  can  quickly  come  to  grips  with  the 
vision 
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g  Employees  for  Action 


Major  organizational  transformations  rarely  happen 
unless  many  people  assist 

Employees  generally  won’t  help  if  they  feel  relatively 
powerless 
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Short-Term  Wins 


•  Major  changes  take  time 

•  People  need  to  see  convincing  evidence  that  the  effort 
is  paying  off 


•  Focus  on  short-term  wins  raises  the  urgency  level  and 
ties  the  transformation  effort  to  the  vision  and  strategy 
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ng  Gains/Creating  More  Change 


If  the  sense  of  urgency  is  lowered,  critical  momentum 
can  be  lost  and  regression  follows 

Irrational  and  political  resistance  to  change  never  fully 
dissipates 


•  Avoid  the  temptation  to  “take  a  break” 

•  Leadership  must  keep  a  long  term  focus  on  the  vision 
and  anticipated  results 
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New  Approaches  in  the  Culture 


The  goal  is  to  permanently  change  the  organization’s 
shared  values 

Cultural  changes  come  last,  not  first 

Cultural  norms  are  many  times  difficult  to  change 

Cultural  shared  values  are  extremely  difficult  to  change 

Will  the  transformation  effort  transcend  any  particular 
individuals??? 
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and  Expanded  Features 


-  Leverages  organizational  transformation  principles 

-  Allows  for  senior  management  prioritization  of  engineering 
system  implementation 

-  Facilitates  organizational  buy-in  and  cooperation 

-  Encourages  cross-organizational  communication 

-  Reduces  resistance  of  engineering  system  adoption  through 
rewards  based  on  independently  verifiable  achievement  of 
management’s  expectations 

-  Allows  management  visibility  into  the  use  of  the  franchisable 
engineering  system 
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Transformation  Infrastructure 


Executive  Vision, 
Direction,  Priorities 


Management 
Steering  Group 


Organizational 
Implementation, 
Standards,  Training 


f 


Project  1 
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Project  2 


Engineering 
Process  Group 


Projects  Implement 
Organizational  Standards 
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Project  n 
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Quality 

Assurance 


Verifies  Implementation  and 
Adherence  to  Standards, 


Reports  to  MSG,  EPG 
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tting  the  Stage 


bstaDiisn  bxecutive  Sponsorship  with  the  expectation  it  is 
active,  not  passive 


2.  Clearly  tie  the  effort  to  business  goals 

3.  Establish  a  guiding  coalition  (MSG/EPG)  of  movers  and 
shakers  from  across  the  organization  to  drive  the  strategy, 
approach,  and  plan 


4.  Projectize  the  effort,  assign  a  cost  center,  and  treat  it  like  a 
project  with  clear  milestones  and  reviews 

5.  Conduct  a  comprehensive  process,  project,  peisonnel,  and 
financial  appraisals  to  establish  an  organizational  baseline 

6.  Tie  implementation  &  adoption  objectives  to  each  individuafs 
performance  review 
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tablishing  the  System 


bstaDiisn  a  measurement  capability  early,  but  don’t 
overwhelm  projects  with  data  gathering  requirements 

Establish  QA  early  to  help  guide  and  mentor,  and  to  report 
engineering  system  adoption  progress 


9.  Ensure  project  schedules  going  forward  contain  all  the 
required  elements  to  meet  the  effort’s  objectives 

1 0.  Either  adopt  processes  &  tools  that  meets  your  needs,  or 
have  the  EPG  design  ones  that  are  better  suited 

1 1 .  Projects  tailor  the  franchise  prototype,  use  them,  and  begin 
performing  better! 

12.  Continue  to  monitor  key  business  measures,  execute  QA,  and 
conduct  senior  management  reviews  to  drive  uigency. 
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The  outcome  will  be  an  integrated,  organizationally 
cooperative  infrastructure  that: 


-  developed  and  deployed  a  franchised  engineering  system 

-  is  the  foundation  for  a  successful  organizational 
transformation 


-  facilitates  engineering  system  improvement  based  on 
consensus  priorities 

-  provides  an  environment  that  supports  project  buy-in  and 
adoption  of  improvements 

-  communicates  effectively  across  the  organization 

-  reports  results  to  senior  management 
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Fast  Track  to  Higher  CAAMI  Maturity  Levels: 
Lessons  Learned  from  Five  Initiatives 


Cheryl  White 
Change  Delivery  Group 

Presented  to:  NDIA 


CMMI  Technology  Conference  and  Users  Group 

November  15,  2007 


Overview 


■  Whether  your  organization  is  Level  1,  Level  5  or 
someplace  in  between,  achieving  higher  CMMI 
maturity  levels  is  often  a  major  investment  in  time, 
capital  and  other  resources 

■  Realizing  an  acceptable  rate  of  Return  on 
Investment  (ROI)  often  depends  on  accelerating 
the  speed  at  which  new  processes  can  be 
implemented  and  adopted 

■  Here's  how  five  organizations  in  commercial, 
government  and  outsourcing  sectors  outperformed 
industry  benchmarks  to  increase  process  maturity  in 
a  remarkably  short  time 
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Five  Case  Studies 

Successes 

Common  Practices 
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Reality 


Historically,  75-857o  of  all  organization 
transformation  initiatives  fail  in  whole  or  in 

part  to  deliver  promised  business  benefits 


For  over  25  years  studies  confirm  this. 

Most  recent  studies:  ProSci  (2006),  US  Army  (2007)  and  IBM  (2006) 
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Outlook  for  the  Year  Ahead. . . 


Each  year,  executives  of  approximately  90%  of 
fortune  500  companies  will  undertake  a  business 
initiative  that  requires  organization  change 

Some  of  these  will  be 
CMMI  initiatives 

Less  than  25%  of  these 
projects  will  show  a 
return  on  investment 
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The  Problem  is  . . . 


Although  Change  Management  and  Business  Process 
Improvement  have  been  around  for  more  than  40 
years,  overwhelming  evidence  suggest  that 


methods  based  on 


Learning 


Sllniulu3.1ar  Ctienge 


these  models  simply 
aren't  reliable 


Diagnosing 


There  must  be  a 


better  way 


Establisliing 
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Alternative  Methods 


By  using  breakthrough  process  improvement 
methods,  these  projects  beat  the  odds. 


Here  is  what  you  can  do  to  increase  the 
success  rate  of  your  next  project 
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Five  Case  Studies 
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The  Projects 


Commercial  Sector 

■  3  SE  organi7ations  within  one  IT  department 
totaling  450  people,  1  VP,  6  directors,  19  projects 

Outsourcing  Sector 

■  2  organizations  within  a  90  person  outsourcing 
facility  providing  IT  development  services  to  the 
insurance  and  US  defense  industries 

Federal  Government 

■  1  project  within  an  agency  of  the  federal 
government 


AH  projects  faced  high  risks 
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Project  profiles 


Case  Study  1:  Commercial  Sector 

■  Initial  assessment  as  LI 

■  Given  2  years  to  assess  as  L2 

■  6  Change  resistant,  hostile  project  teams,  demoralixed 
management 

■  Previous  consultant  asked  to  leave  due  to  non¬ 
performance 

■  18  months  into  corporate  project 

■  Committed  internal  resources 

■  Dwindling  budget 
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Project  profiles 


Case  Study  2:  Outsourcing 

■  Initial  assessment  as  LI 

■  Given  6  months  to  assess  as  L3  (Scampi  Class  B)  by 
major  client  or  lose  contract 

■  Highly  committed  management 

■  No  internal  resources  available 

■  Limited  budget 
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Project  profiles 


Case  Study  3:  Agency  of  Federal  Government 

■  Initial  assessment  as  LI 

■  Need  to  make  changes  to  comply  with  periodic  GAO 
audits 

■  Leadership  focus  directed  to  other  mission  critical 
issues 

■  Initial  lack  of  progress  due  to  general  lack  of  interest 

■  Small  team  of  internal  change  agents 

■  Assisted  by  external  consultants 
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What  Went  Right 
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Summary  of  Outcomes 


Case  Study  1: 

-  LI  to  L3  in  18  weeks  (Scampi  B) 

-  Assessed  as  L3  11  months  from  project  start 
(Scampi  A) 

Case  Study  2: 

-  LI  to  L3  in  14  weeks  (Scampi  B) 

-  Assessed  as  L3  9  months  from  project  start 
(Scampi  A  assessment  grouped  with  other 
organixations) 

Case  Study  3: 

-  Continuous  process  improvement  (validated  by  GAO 
audit) 

-  Date  of  L2  rating  uncertain 
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Comparison  of  Project  Outcomes 


Elapsed  time  to  Sustainable  Process  Maturity 
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Case  Study  1:  Project  Methodology  vs  Other 
Approaches  Used  During  Initiative 


Elapsed  time  to  Sustainable  Process  Maturity 
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OD  Intervention  with  Change  Management 
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Case  Study  1:  Project  (after  6  Months)  vs  Total 
Organization  After  24  Months  (L2  only) 


100% 


REQM  PP  PMC  MA  SAM  CM  PPQA 


■  Affected  organization  of  350  SEs 

■  All  IT  groups  in  1700  person  corporation  including  target  group 
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Common  Practices 
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Common  Practices 


■  Understood  Risk 

■  Focus  on  what  works  best  here 

■  Best  performance  and  local  best  fit  rather  than  on 
global  best  practices 

■  OSSP  reverse  engineered  from  multiple 
instantiated  PDSPs 

■  Non-project  work  performed  by  consultants  so  the 
“real  work"  of  business  could  go  on  throughout  the 
transformation  period 
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Common  Practices 


1  Use  of  standard  methodology  designed  for  Rapid 
Acceleration  of  Change 


Diagnose  Operating  Constraints 


Input 


<o>i 


Output 


Identify  business  &  cultural  obstacles 
to  achievement  of  desired  business 
objectives 


Change  Delivery  Life  Cycle 


Define  Rules 


Input  "VJ 


Assess  business  fit  culture 
performance;  create  gap 
analysis 


Deliberate  on  Delivery 


I  nput  Outp  u  t 


Chunk  change:  prioritize  plan 
incremental  delivery  of  culture  & 
process  improvement;  define  metrics 


Deliver  Change 


Iv 


Input 


Output 


Conduct  18  week  change 
delivery  program 


CMMI  Change  Agents  who  would  never  develop 
software  without  a  POSP  frequently  attempt 
organization  change  without  a  quantitatively  proven 

transformation  process 
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Common  Practices 


2.  Culture  Change  concurrent  with  Process  Change 
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Common  Practices 


3  Organization  training  on  how  to  reengineer 
corporate  culture 


4  Cultural  assessments  occur  throughout  the  process 
improvement  process 

Culture  coaching  helped  teams  overcome 

barriers  to  change 
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Common  Practices 


5  Multi-threaded,  iterative  implementation  cycles 
matched  to  the  organization's  natural  change 
cycles 


^  Organization 
Phase  1 

Change  Rejection 
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Change  Delivery  Life  Cycle 
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Until  Next 
Project  y 

_^Change 
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I  Unchanging 


,  Change 
Rejection 


. Transformation  Module 


Phases  not  to  scale 


J4  KPAs  were  institutionalized  in  18  weeks 
(or  less)  once  planning  was  complete 
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Accelerating  the  Pace  of  Change 
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Critical  Success  Factors 


1  Expected  corporate  benefits  aligned  with  actual 
CMMI  benefits 

2  Leadership  was  stable  and  remains  engaged 
throughout  initiative 

3  One  qualified  consulting  group  led  the  change 
initiative 

4  Consulting  group  had  ready  access  to  leadership 
throughout  program 

5  Core  transformation  team  was  trained  on  methods 
&  tools  used  for  culture  change 

6  4-10%  organization  work  effort  was  committed  to 
transformation  activities 
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Contact  Us  for  more  information  on  these 
and  other  projects 

Change  Delivery  Group 
303.680.0895 


www.changeperfect.com 


Tips  to  Accelerate  Pace  of  Change 


Design  business  rules  to  be  used:  Understand  the  constraints  of  organization 
culture  on  employee  behavior  and  design  new  business  rules,  processes,  and 
technology  to  accommodate  those  constraints 

Limit  disruption  to  business:  When  it  is  a  choice  between  business  as  usual  and 
organization  change,  business  always  wins.  Minimize  disruption  by  implementing 
changes  in  tiny  chunks 

Include  the  right  people  on  your  team:  Some  people  are  keepers  of  culture. 
They  can  tell  you  "what  works  around  here".  Listen  to  them 

Understand  the  comprehension  of  your  sources:  Typically,  people  who  work  in 
organizations  do  not  explicitly  understand  the  basic  rules  of  culture  or  how 
culture  encourages  them  to  behave.  Success  depends  on  knowing  more  about 
culture  than  employees  do 

Design,  develop  and  implement  agilely:  Organization  culture  is  constantly 
changing.  Tap  Into  this  “native"  change  ability  to  propel  your  project  to  success 

Minimize  negative  culture  responses  during  implementation:  Small  bits  of 
change  delivered  incrementally  over  time  cause  less  change  resistance  than 
larger  chunks 
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Tips  to  Achieve  Strategic  Goals 


28 

November, 2004 


Apply  culturally  reinforcing  techniques:  Culture  will  not  push  back  when 
processes  and  technology  support  the  status  quo  by  conforming  to  existing 
organization  rules 

Limit  your  stay  inside  the  organization  and  work  fast:  Organizations 
tolerate  outsiders  temporarily  and  attacks  outsiders  who  refuse  to  comply. 
Most  change  agents  are  Immune  to  attack  for  6  months.  After  that  they 
either  leave  or  they  become  an  agent  of  culture  (rather  than  an  agent  for 
change) 

Be  suspicious  of  corporate  rule  books:  Although  culturally  sanctioned 
behavior  is  pervasive  and  persistent,  it  Is  rarely  documented.  Most  rule 
books  document  behaviors  management  wishes  were  present  and  want  to 
enforce 

Understand  employee  motivation:  Persistent  behaviors,  especially  crazy, 
dysfunctional  or  destructive  behavior  continues  because  culture  rewards 
them 

Be  wary  of  initiatives  under  new  management:  New  managers,  especially 
those  brought  In  to  run  a  change  program,  often  leave  within  2  years. 
(Average  time  In  position  Is  21  months).  Plan  your  project  accordingly 
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Presenter  Bio 


Ms  White  is  a  business  enterprise  architect  specializing  in  the 
design  and  rapid  implementation  of  IT  and  corporate 
transformation  programs.  With  over  20  years  experience  in  a  wide 
range  of  organization  transformation  projects  she  has  led 
strategic  engagements  resulting  In  the  rapid  Implementation  of 
CMMI,  agile  software  development  methods,  ISO  and  sIx-sIgma. 
She  Is  the  author  of  Change  on  Demand:  The  Science  of  Turbo 
Charging  Change  in  Millennium  Corporations  (2007). 
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ADescribe  the  CM  Baseline  Management  process  and  how  it 
relates  to: 

l'  CMMI 

■  ■  _  _ 

I  Program  Execution 

■  ■  _ 

I  Baseball 

ADescribe  the  consequences  of  poor  Baseline  Management 
performance 
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Raytheon 

Space  and  Airborne 
Systems 

Axhe  following  terms  are  used  in  a  generic  manner: 

I  Baseline:  An  approved  work  product  at  a  specific  revision/version  and  date. 

A  baselined  work  product  is  one  that  is  released  and  controlled  by  CM. 

■  ■  _ 

I  Configuration  Baseline:  A  set  of  one  or  more  baselined  work  products 
which  represent  the  approved  version  of  a  predefined  collection  of  work 
products. 

■  ■  _  _  _ 

I  Change  Request  (CR):  A  request  to  change  a  baselined  work  product.  The 

CR  on  programs  could  be  an  PCR,  EO,  SCR,  SPCR,  STR,  etc. 

■  ■  _  _  _ 

I  Configuration  Control  Board  (CCB):  The  board  that  reviews  and 
dispositions  CRs  against  baselined  work  products.  The  board  that  performs 
this  function  could  be  called  any  one  of  a  number  of  names  T  ERB,  CRB, 
SCCB,  CCB,  PRB,  etc. 


CMMI 
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Space  and  Airborne 
Systems 


AConfiguration  Management  (CM)  is  a  process  that 
establishes  and  maintains  the  integrity  of  work  products. 

AConsists  of  five  functional  areas: 

■  ■  ■  ■ 

I  Planning  I  How  will  CM  be  performed  on  a  project? 

I  Configuration  Identification  V  How  will  configuration  items  be 
established  and  work  products  identified  and  what  are  their  relationships 
within  a  product  structure? 

I  Configuration  Control  \  How  will  the  work  products  and  changes  to  the 

work  products  be  controlled? 

■  ■  ■  ■ 

I  Status  Accounting  I  How  will  the  status  of  the  CM  processes  and  program 
work  products  be  managed  and  communicated? 

I  Reviews  &  Audits  I  How  will  the  establishment  and  use  of  the  CM 
processes  be  verified?  How  will  the  control  of  work  products  be  verified? 


CMMI 
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What  is  a  baseline?  What  does  it  . ► 

mean  to  “baseline”  something? 
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/ 
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Defining  Baselines 


Identifying  Baselines 


How  do  I  change  what’s  in  a  baseline? 


V. 
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What  changed  since  yesterday? 
last  year?  last  baseline? 


V 
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Why  should  I  believe  the  CM 
system? 


What  baselines  are  needed  on 
my  project? 


► 


► 


► 
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Controlling  Baselines 


Status  Accounting 
of  Baselines 


Reviews  &  Audits 
of  Baselines 


Planning  Baselines 


CMMI 


What  is  a  baseline  &  why  do  we  have  to  manage  it? 
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Aindividual  work  products 

I  Baseline  nthe  verbo 

■  For  individual  work  products,  the  act  of  releasing  a  work  product  into  the 
configuration  management  system. 

I  Baseline  nthe  nounO 

■  The  version  or  versions  of  the  work  product  in  the  configuration  management 
system. 

AConfiguration  Baseline 

■  ■  _  _ 

I  Common  Configuration  Baselines  include  the  Functional,  Allocated,  and 
Product  Baselines. 


CMMI 
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A  Individual  work  products  have  identifiers 

I  drawing  number 
I  document  ID 
I  code  file  version  number 
A  ...and  revision  or  version  indicators 

I  revision  letter  (e.g.,  Rev.  A) 

I  version  number,  e.g.,  Version  1.2) 

A  Configuration  Baselines  also  have  an  identifier  and  a  revision/version 
indicator 

I  Facilitates  capture  of  different  versions  or  snapshots  of  the  collection  as  the  work 
products,  which  comprise  the  collection,  change 

I  The  CM  information  system  should  provide  the  status  of  a  Configuration  Baseline 
at  selected  points 

■  by  date 

■  software  build  number 

■  hardware  serial  number 


CMMI 


Individual/Configuration  Baselines  must  be  identified 

to  be  effectively  managed. 
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AAn  activity  or  event  triggers  a  work  product  release 

■  ■  ■  ■ 

I  Preliminary  Design  Review  I  Requirements 

I  Critical  Design  Review  -  Design 

APor  Initial  Baseline: 

■  ■  _ 

I  The  baseline  is  audited  to  defined  criteria  for  the  type  of  work  product 
■  ■  _ 

I  The  configuration  records  and  references  are  created  in  the  CM  system 
■  ■  _ 

I  The  baseline  is  released  in  the  CM  System 

■  ■  _  _  _ 

I  The  Configuration  Baseline  is  established  as  identified  in  the  CM  Plan 

APor  Changing  Baselines: 

■  ■  _  _ 

I  Evolving  baselines  are  maintained  in  the  CM  System  as  the  CCB  authorizes 

changes  to  be  incorporated  into  new  versions  of  work  products  and 

Configuration  Baselines. 


CMMI 


Baselines  are  established  and  evolve  in  the  CM  System 
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Ain  a  CMMI-compliant  CM  process,  baselines  are 

I  Created  (CM  SP  1.3) 

■  Authorized  by  an  approval  board  (e.g.,  CCB) 

■  Using  controlled  items  in  the  CM  system 

■  Identified  in  the  CM  System,  including  the  current  configuration  baselines 

■  ■ 

I  Managed 

■  Using  specific  baseline  processes  (CM  GP  2.2,  3.1) 

■  Within  an  established  CM  System  (CM  SP  1.2) 

■  Controlled  changes  to  baselines  (CM  SP  2.2) 

I  Verified 

■  Audited  baselines  as  theyae  established  (CM  SP  3.2) 

■  Audited  controlled  baselines  using  CM  records  (CM  SP  3.1,  GP  2.9) 


CMMI 


Good  CM  processes  include  Baseline  Management 
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AThere  are  parallels  between  good  Baseline  Management  and 

winning  at  baseball 

■  ■ 

I  With  a  more  mature  understanding  of  processes  and  mature  products  (work 
products/players)  it  is  easier  to  be  successful  (stable  baselines/home  runs) 

■  ■ 

I  Both  have  recognized  industry  standards 
■  ■  _ 

I  Team  members  must  work  together  to  be  successful 

■  ■ 

I  New  technologies/players  can  go  through  a  try  out  period  to  identify 
strengths  and  areas  to  develop.  For  companies,  this  evolving  set  of  work 
products  are  a  company  asset  and  should  be  baselined  and  managed. 

■  ■ 

I  Good  management  is  essential  to  being  successful 

■  Day  to  day 

■  Long  term 


CMMI 
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AThe  following  topics  illustrate  the  similarities  between  the 
Baseline  Management  process  and  Baseball: 

I  Individual  Baseline 
I  Baseline  Verification 
I  Configuration  Baseline 
I  Product  Baseline 
I  Opponents 
I  Results  of  Winning 


CMMI 
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I  Identify  Work  Product 
I  Create  Work  Product 
I  Successful  Peer  Review 
I  Successful  CCB  Review 
I  Release  (Baselined)  Work  Product 

A  Comments 

I  Unless  the  Work  Product  is  created  (player  able  to  advance  to  First  Base),  the 
process  cannot  begin 

I  Unless  its  Peer  and  CCB  reviewed  and  approved  it  canQ  advance  to  release 

I  There  are  legitimate  ways  to  advance  when  the  ball  isnQ  in  play  (stealing);  however, 
not  following  the  process  creates  problems  (youQe  out!) 

I  Status  Accounting  data  about  Individual  Baselines  are  similar  to  a  playerQ  statistics 
T  how  it  evolved  and  performed  from  inning  to  inning. 


A  Baseball 

Identified  player  at  bat 
Player  at  First  Base 
Player  at  Second  Base^ 
Player  at  Third  Base 
I  Player  at  Home  Plate  (Score) 
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^^Home  Run”  occurs  when  all  steps  are  conducted  smoothly 
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Note:  No  procedures  for  collaboration/sub-contracting  of  products/services  only  for  COTS 
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-1- 


•  SG1  -  Establish  Supplier  Agreements 

-  SP  1.1  -  Determine  Acquisition  Type 

•  Acquisitions  may  be  COTS  from  third-party  vendors,  components 
developed  by  internal  or  external  partner,  or  services  delivered  by 
internal  or  external  partner 

-  SP  1 .2  -  Select  Suppliers 

•  Establish  criteria  for  selection  of  partners  and  also  list  of  preferred 
suppliers/collaboration  partners 

-  SP  1 .3  -  Establish  Agreements  with  Suppliers 

•  Establish  formal  agreements  with  suppliers  and  collaboration 
partners  (service  agreements,  product  development  agreements, 
license  agreements,  etc) 

•  For  internal  partners  the  formal  Supplier  Agreement  is  a 
Collaboration  Plan,  which  is  part  of  the  Project  Plan 
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ocess  Area  -2- 


•  SG2  -  Satisfy  Supplier  Agreements 

-  SP  2.1  -  Execute  the  Supplier  Agreement 

•  For  internal  partners  the  formal  Supplier  Agreement  is  a  Collaboration 
Plan,  which  is  part  of  the  Project  Plan 

-  SP  2.2  -  Monitor  Selected  Supplier  process 

•  For  internal  collaboration  partners  use  internal  release  process 

-  SP  2.3  -  Evaluate  Selected  Supplier  Work  Products 

•  This  applies  to  internal  developed  components  or  services  such  as  testing 

-  SP  2.4  -  Accept  the  Acquired  Product 

•  Services  such  as  testing  are  also  considered 

-  SP  2.5  -  Transition  Products 

•  Services  such  as  testing  are  also  considered 
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Verification  as  a  Service 


Activity  - 


•  System  and  Integration  Testing  considered 
as  a  Service  Delivery  activity  in  the 
organization 

•  SAM  not  Implemented  in  the  past  in  the 
organization 

-  Service  Delivery 

-  Capacity  and  Availability  Management 

-  Problem  Management 

-  Incident  and  Request  Management 
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bell  I  ipic  ouTT^boration  Agreement  Templates 


•  Sample  Templates  derived  from  SAM  PA  to  be 
distributed  and  discussed  with  attendees: 


-  Collaboration  Agreement  for  Remote  Product 
Development 

-  Collaboration  Agreement  Template  for  Remote  Service 
Delivery  (Software  TestingA/erification) 
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MA  Process  Area 


•  Measurement  Objective 

-  To  improve  “partner’s”  satisfaction 

•  Measures 

-  Number  of  “partner’s”  complaints 

•  Party  or  stakeholder  involved  in  collaboration  can  enter  a 
complaint  after  a  week  of  not  having  received  response  to  an 
issue 

-  Level  of  severity  of  “partner’s”  complaints 

•  Low  -  first  entry  associated  with  a  complaint 

•  Medium  -  second  entry  associated  with  a  previous  complaint 

•  High  -  more  than  two  entries  associated  with  a  previous 
complaint 


19 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


MA  Process  Area 


Data  Collection  and  Storage 


A  partner/stakeholder  enters  a  written  complaint  in  the 
Complaint  Spreadsheet  available  in  the  Project  Common 
repository 

The  Complaint  Spreadsheet  has  several  sections  each 
regarding  the  identified  type  of  collaboration 

The  complaints  are  reviewed  weekly  at  the  Senior 
Management  meetings 

Each  manager  is  responsible  to  ensure  any  complaints  are 
properly  addressed 

Complaint  Spreadsheet  is  maintained  by  Director  of 
Development  under  CM 
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MA  Process  Area 


Analysis  of  Measurement  Data 


-  Histogram  showing  number  of  complaints  clustered  by 
severity  level  are  developed  by  Director  of  Development 
Solutions 
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MA  Process  Area 


•  Reporting  of  Measurement  Data 


-  Histogram  charts  are  presented  at  the  end  of  each  month 
and  discussed  at  the  Senior  Management  Meeting 

-  Any  corrective  actions  are  tracked  to  completion  by 
Director  of  Development  Solutions 
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SaMiiTie  bf  Complaints  Sheet  2Q 

of  2006 


Com|}liiiiit° 

Severity 

Level 

Description 

Requestor 

Party 

Requested 

Party 

Date  Created 

Date  Expected 
Resolution 

Comments 

Date  Resolved 

Date  Second 
Entiy 

Date  Second 
Entiy 

1 

L 

More  detailed  description  of  Requirement  DS 
0012  and  no  response  from  Development 
qroup  in  US 

RSiD  India 

R&D  US 

6-Jul-06 

13-Jul-06 

Details  were  obtained  from  Product  Manager  on 
12-Jul-06-2006 

12-Jul-06 

2 

H 

Provide  details  about  Performance 
Requirements  and  no  response 

Test  Group 
India 

Product 

Mqmt 

6-Jul-06 

13-Jul-06 

Request  passed  by  Director  to  Product  Manager 
but  no  response  as  of  13-Jul-06. Second  request 
by  Director  to  Product  Manager  on  20-Jul-06  and 
no  answer. 

26-Jul-06 

13-Jul-06 

20-Jul-06 

3 

M 

Review  of  general  architecture  document  for 
Dashboard  module  without  any  response 

R&D  India 

R&D  US 

12-Jul-06 

19-Jul-06 

Due  to  lack  of  time  Dashboard  Architecture 
Document  was  not  reviewed  until  July  28  of  2006 

28-Jul-06 

19-Jul-06 

4 

M 

Give  presentation  on  new  requirements  on 

MRD  without  any  response 

RSiD  India 

Product 

Mqmt 

13-Jul-06 

20-Jul-06 

Director  remindeded  Product  Manager  to  give 
presentation  to  R&D  Group  in  India.  Product 
Manager  responded  on  July  21  of  2006. 

26-Jul-06 

20-Jul-06 

5 

L 

Provide  feedback  on  Test  Plan  and  no 
feedback  or  notice  received  yet 

Test  Group 
India 

R&D  US 

20-Jul-06 

27-Jul-06 

RSiD  qroup  will  review  test  Plan 

25-Jul-06 

6 

7 

8 

9 

10 

11 
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Sample  Histogram  of  Complaints 


2Q  Complaints  -  R&D  India  to  R&D  US 


High  Medium 

Complaint  Type 


Low 
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Conclusions 


1 


•  Geographically  dispersed  teams  at  SAS: 

-  Product  Development 

-  System/Integration  Testing 

•  System  /Integration  Testing  viewed  as  Service 

•  With  a  low  number  of  distributed  projects,  an  informal 
method  to  collaborate  was  sufficient 


•  SAM  CMMI  PA  needed  as  number  of  projects  increased 

•  The  practices  of  the  SAM  CMMI  process  area  are 
successfully  being  used  to  manage  both  remote  product 
development  and  service  delivery 
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Conclusions 


•  Including  the  CMMI  MA  PA  helps  monitoring 
effectiveness  of  process 

•  Essential  to  build  a  lean  process 

•  Focusing  on  the  “most  painful”  areas  was  important  for 
buy-in 

•  Use  of  SAM  process  reduced  level  of  frustration  in 
remote  “sister”  organizations 

•  Resistance  on  process  came  from  “responsible”  partner 

•  Use  of  templates  facilitated  implementation  of  SAM 
process 


•  Metric  was  identified  by  members  of  the  development 
and  testing  organizations 
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Agenda 


•  Background 

•  Challenges 

•  Solution:  Automated  Management  Systems 

•  Automated  System  Toolset 

-  Project  Planning  and  Scheduling 

-  Technical  Performance  Management 

-  Earned  Value  Management 

-  Risk  Management 

-  Resource  Management 

-  Defect  management 


Background 


•  Global  Computer  Enterprises  (GCE) 

-  Systems  Integration  Organization 

-  Federal  Government  Contractor 

-  CMMI 

•  Level  3  Certified  Organization 

•  Pursuing  Maturity  Level  4 

•  Projects  Managed 

-  Various  Government  Agencies 

•  General  Services  Administration  (GSA) 

•  Department  of  Defense  (DOD) 

•  United  States  Coast  Guard  (USCG) 

•  Transportation  Security  Administration  (TSA) 

•  Domestic  Nuclear  Detection  Office  Organization  (DNDO) 

•  United  States  Secret  Service  (USSS) 

-  Firm-Fixed-Price  Contracts 

-  Project  portfolio  for  each  Agency  or  program  within  the  Agency 

-  Delivering  Earned  Value  Management  for  all  projects 
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Project  Portfolio  Management 


] 


•  Project  Portfolio  Management  (PPM)  is  a  management  approach 
characterized  by  treating  related  projects  as  part  of  an  overall 
project  investment  portfolio 

•  PPM  establishes  a  set  of  values,  techniques  and  technologies  that 
enable  visibility,  standardization,  measurement  and  process 
improvement  across  all  projects 


PPM 

Software  Development  &  Integration 

Project  Portfolio 

Project  /  Product  Release 

Project  Investment 

Project  /  Deliverable 

PPM  Challenges 


Management  Process 

Challenges 

Project  Portfolio  Management 

Repeatable,  integrated  execution  of  all  the  management  processes 

Project  Planning  and 

Scheduling 

Work,  task  breakdown  across  overlapping  projects  and  shared  resources 
Keeping  track  of  constant  schedule  changes 

Technical  Performance 
Management 

Micro  level  work  assignment  and  tracking  is  time  consuming 

Status  checking  involves  intensive  floor  management 

Earned  Value  Management 

Collecting  EVM  data  is  labor  and  time  intensive 

Involves  perusing  different  documents  such  as  project  plans,  status  reports 
spread  across  documents  and  excel  sheets 

Risk  Management 

Tracking  cost  and  schedule  performance  while  taking  risks  into 
consideration  is  an  added  complexity 

Resource  Management 

Resource  utilization  to  obtain  real-time  project  costs  and  resource  pipeline 
Management 

Defect  Management 

Integrated  defect  detection  and  resolution  of  defects  in-place  during  the 
course  of  the  projects 

Business  Intelligence 

Generating  status  reports,  obtaining  measures  and  quantitative  information 
for  a  collection  of  projects  is  a  tedious  manual  process 

Solution:  Automated  Management  Systems 


Management  Process 

Solution 

Project  Portfolio  Management 

Automated  System  to  implement  and  support  these  management  processes 

Project  Planning  and 

Scheduling 

Planning  with  EVM  emphasis  in  mind 

Predefined  and  customizable  Work  Breakdown  Structure  and  Work 

Distribution  Structure  in  the  system 

Technical  Performance 
Management 

Robust  Management  of  tasks 

Task  management  and  workflow  to  transition  tasks 

Task  Inbox  for  each  project  team  member 

Real-time  status  report  on  overall  project  progress 

Earned  Value  Management 

EVM  data  obtained  from  the  collective  repository  of  projects,  tasks,  work- 
items  and  activities 

Financial  Controls 

Early  Warning  mechanisms 

Risk  Management 

Integrated  Risk  tracking  and  Risk  life  cycle  management 

Resource  Management 

Timesheet  functionality  integrated  with  task  logging  against  the  work 
Breakdown 

Defect  Management 

Defect  collection,  tracking  and  integrated  defect  resolution  task  management 

Business  Intelligence 

Obtained  from  the  collective  repository  of  project  management  data 

E.g.  generate  real-time  EVM  reports,  productivity  measures 

Automated  System  Toolset 

Selection  Criteria 


-  Automated  Processes 

-  Open  Source  Systems 

-  Integrated  to  manage  technical,  schedule,  and  cost 
performance 

-  Scalable,  customizable  and  extensible 
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System 


Schedule  Management 


Tool 


Dotproject 


Task,  Cost  and  Timesheet  Management 


Dotproject 


EVM  Data  Repository 


MySQL  Database 


EVM  Reports 


Informatica 


Early  Warning  System 


Php  extensions 


Alerts 


Postfix 


Defect  Management 


Dotproject,  JIRA 


Project  Planning  and  Scheduling 


•  Project  plans  are  developed  with  an  emphasis  on  EVM 

•  Work  Breakdown  structure 

-  Based  on  PPM 

-  Adopt  iterative  development  model 

-  Agile  practices 

-  Granularity:  Estimate  atomic  task  assignments  at  hourly  level  of  detail 

•  Work  Distribution  structure 

-  SDLC  based 

•  Distribution  across  SDLC  phases 

-  Role  based 

•  Resource  assignment  by  segregation  of  duties 

-  Dependencies  recorded  and  tracked 


Work  Breakdown  Structure 

1  1  1 

1 

- ^  - 1 - 

'1 

i 

"i 

j 

L 

_ 

■ 
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Technical  Performance  Management 


•  Online  Work  Management  System  (WMS) 

-  Web-based  project  management  tool 

-  Robust  portfolio  management  of  projects  and  micro  tasks  for  all 
organization 

-  Monitor  and  track  all  projects  and  tasks 

•  Real-time  Tracking 

-  Project  actual  %  completion  available  real-time 

•  Independent  assessment 

•  Objective  evidences 

-  Ability  to  monitor  project  progress  in  real  time 

-  Slice  and  dice  data  across  releases,  deliveries  and  projects 

•  Task  Life  Cycle  Management 

-  Online  task  creation,  assignment  and  completion 

-  Task  status  reporting  of  complete,  pending  tasks 


Technical  Performance:  Portfolio  Status 


Progress 


Project  Name  Start  Date  End  Date  Owner 


60.07o  I 

75.07o  I 

100.0%  I  smKjiE)  LcffJtseo^i  r  q  w 

96.97o  I  i^0DE{]>KK0MijlSia<lldnti^lffilFl334ljflk 
100. 07o  I  Z5»3ia<Ea01dFJt^l3j!ififiy WEBER 


09/13/2007  09/21/2007  SftlfaiMgirttlirbcfaftlifd 
07/09/2007  09/30/2007  OgnSBlgWSPtflkJfa 
09/28/2007  10/01/2007 
09/10/2007  10/05/2007  (Elifrwsiip>  PNipesfrer 
09/29/2007  10/05/2007  WMMcSbbceDiefeliwefle 


Status 
In  Progress 
In  Progress 
In  Progress 
In  Progress 
In  Progress 


High  Level  Portfolio  Status  view 


Technical  Performance:  Project  Status  / 
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Pin 

New 

Log 

Work 

Percent  External  p 

Weiqhtaqe  Assesment 

^^ask  Name 

Line  Of  Business 

SDLC  Phase 

Milestone  in  SDLC  Phase 

Technology  Stack 

^  * 

Log 

1007o 

07o 

07o  1  ^  Service  Pack  11(13) 

Operation 

^  * 

Log 

1007o 

07o 

07o  r 

^  rrepanc)/  m  Warning  Message 

format  causing  issues  (4) 

Operation 

f  * 

Log 

1007o 

07o 

07o  r 

Technical  Resolution 

Maintenance 

Technical  Resolution 

Technical  Draft 

Resolution 

Business 

Services 

^  * 

Log 

1007o 

07o 

07o  r 

=■■■  Development 

Maintenance 

Development 

Business  Logic 

Business 

Services 

* 

Log 

1007o 

07o 

07o  r 

Functional  Certification 

Maintenance 

Development 

Functional  Certification 

Business 

Services 

* 

Log 

1007o 

07o 

07o  r 

Technical  Certification 

Maintenance 

Development 

Technical  Certification 

Business 

Services 

Project  Gantt  view 


Earned  Value  Management 


•  EVM  data 

-  Real-time  data  from  WMS 

-  Estimates 

-  Project  percent  completion 

-  Funds  Burned 

-  Schedule  Burned 

•  Funding  Variance  controls 

-  Automatic  alerts  when  funding  variances  exceed  threshold 

•  Uniform  Spending 

-  Permit  task  performance  and  work  logging  only  within  the  budgeted  weekly  burn 
rate 

•  Task  and  Project  Period  of  performance 

-  permits  task  performance  and  logging  only  with  the  project  period  of  performance 
of  task  or  project 

•  Real-time  Reports 

-  Visibility  into  SPI  and  CPI 

-  Accurate  and  timely  data 

-  Effective  decision  making 
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Real-time  EVM  Report 


Project 

Name 

Period 

Of 

Performance 
(  in  Days  ) 

Funding 

Level 

Scheduled 

Days 

Left 

Total 

Funding 

Left 

Percentage 

Schedule 

Burned 

Percent 

Completed 

Schedule 

Variance 

Percent 

Funding 

Burned 

Funding 

Variance 

Projected 
Earning 
Per  Burn 
Rate 

Actual 

Earning 

Project  1 

91 

$356.25 

52 

$261.75 

42.86% 

30.77% 

26.53% 

4.24% 

$94.50 

$109.62 

Project  2 

91 

$14,207.74 

52 

$10,787.24 

42.86% 

38.46% 

-4.40% 

24.07% 

14.39% 

$3,420.50 

$5,464.30 

Project  3 

91 

$494.00 

52 

$458.00 

42.86% 

33.00% 

7.29% 

25.71% 

$36.00 

$163.02 

Project  4 

91 

$15,547.12 

52 

$13,459.12 

42.86% 

25.51% 

13.43% 

12.08% 

$2,088.00 

$3,966.07 

Project  5 

91 

$4,984.04 

52 

$3,724.04 

42.86% 

38.46% 

-4.40% 

25.28% 

13.18% 

$1,260.00 

$1,916.86 

Project  6 

91 

$1,004.81 

52 

$853.81 

42.86% 

38.46% 

-4.40% 

15.03% 

23.43% 

$151.00 

$386.45 

Project  7 

91 

$1,534.62 

52 

$702.12 

42.86% 

46.15% 

3.29% 

54.25% 

$832.50 

$708.23 

Project  8 

91 

$2,280.00 

52 

$1,272.00 

42.86% 

46.15% 

3.29% 

44.21% 

1.94% 

$1,008.00 

$1,052.22 

Real-time  EVM  Report 


Real-time  EVM:  Early  Warning  Mechanisms. 


•  Calculate  cost  and  schedule  variances 

-  Automated  check  on  each  project 

-  Calculated  from  integrated,  real-time  WMS  system 

•  Identify  work  variance  thresholds 

-  Variances  exceed  acceptable  tolerances 

•  Schedule  burned 

•  Funding  burned 

•  Automated  alerts  when  variance  thresholds  are  exceeded 

-  Program  Management 

-  Execution  Teams 

•  Risk  Management 

-  Identify  cost  and  schedule  overrun  risks  at  an  early  stage 

-  Respond  more  quickly  with  mitigation  strategies 


Risk  Management 

•  Risk  Identification 

-  Risk  details  such  as  probability  and  impact  of  risk 

•  Risk  Analysis 

-  Association  with  a  task  (Origin  of  risk),  actual  impact  (number  of 
days  of  effort,  total  dollars  for  equipment  etc.) 

•  Risk  Mitigation 

-  Planning  changes 

-  Risk  mitigation  tasks  created  and  assigned 

•  Risk  Monitoring  and  Control 

-  Resolution  of  the  risk 

-  Implement  the  tasks  for  containing  the  risk 

-  Tracking  and  communication  of  risk  mitigation  tasks 

-  Budget  and  cost  automatically  updated 
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Resource  Management 


•  Utilization  Reports 

-  Overutilization 

-  Underutilization 

•  Cumulative  timesheet  entries  from  task  logs 

-  Record  and  report  time  worked  on  a  project 

•  Identify  trends 

-  Workload 

-  Resource  management 


Users:  All 

T 

Projects:  All 

T 

Users 

Week  40 

Week  41 

Week  42 

Week  43 

Week  44  Week  45  Week  46  Week  47 

Week  48 

Week  49 

Week  50 

Week  51 

Week  52 

SW  Engineer  ■ 

22.79 

22.79 

22.79 

22.79 

22.79 

22.79 

22.79 

22.79 

22.79 

41.21 

41.21 

22.42 

22.42 

S/W  Engineer 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

SAV  Engineer 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

40 

SAV  Engineer 

30.86 

30.86 

30.86 

30.86 

30.86 

30.86 

30.86 

30.86 

17.33 

17.33 

17.33 

14.73 

14.73 

SAV  Engineer 

26.71 

33.17 

33.17 

33.17 

33.17 

33.17 

33.17 

33.17 

25.32 

25.32 

25.32 

25.32 

25.32 

SAV  Engineer 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

39.57 

Real-time  Resource  Allocations  view 


Resource  Management  Contd. 
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•  Timesheet  is  integrated  within  the  WMS 

-  Report  by  hierarchical  work  breakdown  structure 

-  Report  by  individual  user,  project,  division 


Project/UserName  Sep  23-29 

Sep  30-Oct  06 

Oct  07-13 

Release  1 

1457.08 

1481.27 

1385.5 

Delivery  2 

1457.08 

1481.27 

1385.5 

^■■■Project  1 

91.5 

84 

106.8 

^■■Engineer  1 

21 

0 

32 

^■■Engineer  2 

0 

0 

0 

^■■Manager  1 

27 

40 

40 

:■■■  Architect  1 

0 

0 

12 

^■■QA  1 

23.5 

20 

22.8 

^■■■QA  2 

20 

24 

0 

L.  Project  2 

74 

77 

59.5 

■■■■Manager  2 

32 

33 

28.5 

Engineer  3 

17 

36 

21 

^■■Engineer  4 

25 

8 

10 

^■PrDj&ct  3 

78.5 

91.5 

76 

LCM  1 

27 

28.5 

40 

^■■System  Admin  2 

15 

32 

30 

^■■■DBA3 

36.5 

31 

6 

=  PrDject  4 

16 

4 

20 

Hierarchical  Task  Hour  Report 


Resource  Management  Contd. 


^  Weekly  Time  Card 

[7]  Saturday  10/06/2007  through  Friday  10/12/2007  test  user  (testuser) 

T  [My  Time  Card] 

Task  Name  Task  Log  Type  Log  Entry 

Saturday  10/06/2007 

Hours 

Sunday  10/07/2007 

Total  Hours 

[H 

Monday  10/08/2007 

Total  Hours 

[H 

Tuesday  10/09/2007 

Total  Hours 

[H 

Total  Hours 

[H 

Wednesday  10/10/2007 

Total  Hours 

[H 

Thursday  10/11/2007 

Total  Hours 

[H 

Friday  10/12/2007 

Total  Hours 

[H 

For  the  week  ot  Saturday  10/06/2007  througn  rrioay  lu/i^/^uu/ 

Total  Hours 

Status 

E 

Weekly  Timesheet  Report 


Defect  Management 


•  Integrated  with  the  projects  and  tasks  in  the  WMS 
system 

•  Defect  Tracking 

-  Originating  task 

-  SPR  number  created  in  JIRA 

-  Task  is  executed  through  phases  of  SDLC 

•  Task  Performance  Measurement 

-  Software  defects 

-  Document  issues 

-  Meeting  attendance 

•  Reports 

-  Defect  density 

-  Defects  per  KSLOC 

-  Defect  statistics  by  origin,  project,  resource 


] 


Business  Intelligence 


•  Task  Management 

-  Task  tracking  reports 

-Task  status  reporting  of  compiete,  pending 
tasks 


•  Risk  Management  Measures 

•  Defect  Measures 

•  Resource  Utilization  Measures 


] 


Business  Intelligence  Contd. 


Projects:  Project 33 


Current  Project  Status  Task  Assignee  Pending  Tasks  Overdue  Tasks  In  progress  Completed  Tasks  Total  Tasks  Hours  worked 


Status  Task  Details 

% 

_ 

0 

1 

1 

14 

15 

158  hours 

Complete:  88 

85% 

1 

2 

1 

15 

17 

68  hours 

In  Progress:  1 

17o 

_ _ 

8 

9 

1 

19 

28 

91.5  hours 

Not  Started:  14 

14 

^ . . . 

2 

3 

1 

29 

32 

129.5  hours 

Past  Due:  15 

15% 

0 

1 

1 

6 

7 

47.5  hours 

Total:  103 

1007o 

- - 

0 

0 

0 

19 

19 

76  hours 

0 

0 

0 

0 

1 

0  hours 

Project  Assignee  Details 

Team  Size:  9  users 

0 

0 

0 

1 

1 

0  hours 

_ 

0 

0 

0 

7 

7 

0  hours 

Document  Space  Utilized 

Total: 

14 

14 

1 

88 

103 

570.5  hours 

Space  Utilized:  0  B 


Project  Statistics  Dashboard 


Business  Intelligence  Contd. 


Project  Defects  Dashboard 


Business  Intelligence  Contd. 


] 


Project  Effort  Estimate  Variance  Dashboard 


Tying  it  back  to  CM  Mi 


PPM  Processes 

CM  Ml  Process  Areas 

Maturity  Level 

Project  Portfolio  Management 

Integrated  Project  Management  (IPM) 

3 

Project  Planning  and  Scheduling 

Project  Planning  (PP) 

2 

Technical  Performance  Management 

Project  Monitoring  and  Control  (PMC) 

2 

Earned  Value  Management 

Integrated  Project  Management  (IPM) 

3 

Project  Monitoring  and  Control  (PMC) 

2 

Risk  Management 

Risk  Management  (RSKM) 

3 

Validation  (VAL) 

3 

Defect  management 

Verification  (VER) 

3 

Resource  Management 

Project  Planning  (PP) 

2 

Measurement  and  Analysis  (M&A) 

3 

Quantitative  Project  Management 

4 

Business  Intelligence  Reports  and  Dashboards 

Organizational  Process  Performance  (OPP) 

4 

References 
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-  0MB  Circular  A11 ,  Part  7 
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-  American  National  Standards  Institute  (ANSI)/Electronic  Industries  Alliance  (EIA) 
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1998.  It  was  reaffirmed  on  August  28,  2002. 
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-  PMI  College  of  Performance  Management 

•  http://www.pmi-cpm.ora/paaes/home/index.html 

-  Dotp reject 

•  http://www.dotproiect.net/ 

-  Quantitative  Methods  in  Project  Management 

•  John  C  Goodpasture.  (2004) 

-  Agile  EVM  -  Earned  Value  Management  The  Agile  Way 

•  Tamara  Suleiman 

-  CMMI:  Guidelines  for  Process  Integration  and  Product  Improvement,  Second 
Edition 

•  Mary  Beth  Chrissis,  Mike  Konrad,  and  Sandy  Shrum 


Summary 


•  Automation  leading  to  PPM  approach  easily 
implemented  by  a  smaller  organization 

•  Solution  for  common  PPM  challenges  across  all 
organizations 

•  Automated  PPM  provided  the  foundation 

-  Easier  CMMI  adoption 

-  Levei  3  Appraisai 

•  Intention  to  approach  ML4  activities  in  a  similar  fashion 

•  Thoughts 

-  Real-time  introspective  management  vs.  retrospective 
management 

-  Emphasis  on  forecasting  for  tomorrow  rather  than  project 
instances 


Thank  you 
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•  Describe  the  CM  Baseline  Management  process  and  how  it 
relates  to: 

—  CMMI 

—  Program  Execution 

—  Baseball 

*  Describe  the  consequences  of  poor  Baseline  Management 
performance 


Purpose 


CMMI 
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Common  Terms 
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Systems 


•  The  following  terms  are  used  in  a  generic  manner: 

—  Baseline:  An  approved  work  product  at  a  specific  revision/version  and  date. 
A  baselined  work  product  is  one  that  is  released  and  controlled  by  CM. 

—  Configuration  Baseline:  A  set  of  one  or  more  baselined  work  products 
which  represent  the  approved  version  of  a  predefined  collection  of  work 
products. 

—  Change  Request  (CR):  A  request  to  change  a  baselined  work  product.  The 
CR  on  programs  could  be  an  PCR,  EO,  SCR,  SPCR,  STR,  etc. 

—  Configuration  Control  Board  (CCB):  The  board  that  reviews  and 
dispositions  CRs  against  baselined  work  products.  The  board  that  performs 
this  function  could  be  called  any  one  of  a  number  of  names  -  ERB,  CRB, 
SCCB,  CCB,  PRB,  etc. 


CMM! 
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What  is  Configuration  Management? 
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•  Configuration  Management  (CM)  is  a  process  that 
establishes  and  maintains  the  integrity  of  work  products. 

•  Consists  of  five  functional  areas: 

—  Planning  -  How  will  CM  be  performed  on  a  project? 

—  Configuration  IdentiUcation  -  How  will  configuration  items  be 
established  and  work  products  identified  and  what  are  their  relationships 
within  a  product  structure? 

—  Configuration  Control  -  How  will  the  work  products  and  changes  to  the 
work  products  be  controlled? 

—  Status  Accounting  -  How  will  the  status  of  the  CM  processes  and  program 
work  products  be  managed  and  communicated? 

—  Reviews  &  Audits  -  How  will  the  establishment  and  use  of  the  CM 
processes  be  verified?  How  will  the  control  of  work  products  be  verified? 


CMM! 
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What  is  Baseline  Management?  (1) 

What  is  a  baseline?  What  does  it  . 

mean  to  “baseline”  something? 


Raytheon 

Space  and  Airborne 
Systems 

Defining  Baselines 


What’s  in  a  baseline? 


►  Identifying  Baselines 


How  do  I  change  what’s  in  a  baseline? . Controlling  Baselines 


What  changed  since  yesterday?  ■  . . . . .  ■  Status  Accounting 

last  year?  last  baseline?  of  Baselines 


Why  should  I  believe  the  CM  . ►  Reviews  &  Audits 

system?  of  Baselines 


What  baselines  are  needed  on  . ►  Planning  Baselines 

my  project? 

_ J 


CMMI^ 


What  is  a  baseline  &  why  do  we  have  to  manage  it? 
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What  is  a  Baseline?  (2) 
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•  Individual  work  products 

—  Baseline  “the  verb” 

■  For  individual  work  products,  the  act  of  releasing  a  work  product  into  the 
configuration  management  system. 

—  Baseline  “the  noun” 

■  The  version  or  versions  of  the  work  product  in  the  configuration  management 
system. 

•  Configuration  Baseline 

—  Common  Configuration  Baselines  include  the  Functional,  Allocated,  and 
Product  Baselines. 


CMM! 
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•  Individual  work  products  have  identifiers 

—  drawing  number 

—  document  ID 

—  code  file  version  number 

•  ...and  revision  or  version  indicators 

—  revision  letter  (e.g.,  Rev.  A) 

—  version  number,  e.g.,  Version  1.2) 

•  Configuration  Baselines  also  have  an  identifier  and  a  revision/version 
indicator 

—  Facilitates  capture  of  different  versions  or  snapshots  of  the  collection  as  the  work 
products,  which  comprise  the  collection,  change 

—  The  CM  information  system  should  provide  the  status  of  a  Configuration  Baseline 
at  selected  points 

■  by  date 

■  software  build  number 

■  hardware  serial  number 


CMMi 


Individual/Configuration  Baselines  must  be  identified 

to  be  effectively  managed. 
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How  are  Baselines  Controlled? 
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•  An  activity  or  event  triggers  a  work  product  release 

—  Preliminary  Design  Review  -  Requirements 

—  Critical  Design  Review  -  Design 

•For  Initial  Baseline: 

—  The  baseline  is  audited  to  defined  criteria  for  the  type  of  work  product 

—  The  configuration  records  and  references  are  created  in  the  CM  system 

—  The  baseline  is  released  in  the  CM  System 

—  The  Configuration  Baseline  is  established  as  identified  in  the  CM  Plan 

•  For  Changing  Baselines: 

—  Evolving  baselines  are  maintained  in  the  CM  System  as  the  CCB  authorizes 
changes  to  be  incorporated  into  new  versions  of  work  products  and 
Configuration  Baselines. 


Baselines  are  established  and  evolve  in  the  CM  System 


CMMI 
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CMMI  and  Baseline  Management 
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•  In  a  CMMI-compliant  CM  process,  baselines  are 

—  Created  (CM  SP  1 .3) 

■  Authorized  by  an  approval  board  (e.g.,  CCB) 

■  Using  controlled  items  in  the  CM  system 

■  Identified  in  the  CM  System,  including  the  current  configuration  baselines 

—  Managed 

■  Using  specific  baseline  processes  (CM  GP  2.2,  3.1) 

■  Within  an  established  CM  System  (CM  SP  1.2) 

■  Controlled  changes  to  baselines  (CM  SP  2.2) 

—  Verified 

■  Audited  baselines  as  they’re  established  (CM  SP  3.2) 

■  Audited  controlled  baselines  using  CM  records  (CM  SP  3. 1,  GP  2.9) 


Good  CM  processes  include  Baseline  Management 


CMMI 
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Baseline  Management  and  Baseball  (1)  space  and  Airbom® 

o  V  /  Systems 

•  There  are  parallels  between  good  Baseline  Management  and 
winning  at  baseball 

—  With  a  more  mature  understanding  of  processes  and  mature  products  (work 
products/players)  it  is  easier  to  be  successful  (stable  baselines/home  runs) 

—  Both  have  recognized  industry  standards 

—  Team  members  must  work  together  to  be  successful 

—  New  technologies/play ers  can  go  through  a  try  out  period  to  identify 
strengths  and  areas  to  develop.  For  companies,  this  evolving  set  of  work 
products  are  a  company  asset  and  should  be  baselined  and  managed. 

—  Good  management  is  essential  to  being  successful 

■  Day  to  day 

■  Long  term 


CMM! 
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Baseline  Management  and  Baseball  (2) 
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•  The  following  topics  illustrate  the  similarities  between  the 
Baseline  Management  process  and  Baseball: 

—  Individual  Baseline 

—  Baseline  Verification 


—  Configuration  Baseline 

—  Product  Baseline 

—  Opponents 

—  Results  of  Winning 


CMM! 
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•  Configuration  Management 

—  Identify  Work  Product 
—  Create  Work  Product 
—  Successful  Peer  Review 
—  Successful  CCB  Review 
—  Release  (Baselined)  Work  Product 


Baseball 

—  Identified  player  at  bat 

—  Player  at  First  Base 

—  Player  at  Second  Base^ 

—  Player  at  Third  Base 

—  Player  at  Home  Plate  (Score) 


•  Comments 

—  Unless  the  Work  Product  is  created  (player  able  to  advance  to  First  Base),  the 
process  cannot  begin 

—  Unless  its  Peer  and  CCB  reviewed  and  approved  it  can’t  advance  to  release 

—  There  are  legitimate  ways  to  advance  when  the  ball  isn’t  in  play  (stealing);  however, 
not  following  the  process  creates  problems  (you’re  out!) 

—  Status  Accounting  data  about  Individual  Baselines  are  similar  to  a  player’s  statistics 
-  how  it  evolved  and  performed  from  inning  to  inning. 


“Home  Run”  occurs  when  all  steps  are  conducted  smoothly 

CMlIff* 
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Baseline  Verification 
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•  Configuration  Management 

—  Baseline  Audits 
—  Proeess  and  Produet  Audits 


•  Baseball 

—  Umpires 


•  Comments 

—  Like  baseball,  Work  Product  Baselines  are  verified  as  they  are  established. 

■  Audits  are  performed  on  work  products  prior  to  baseline  (Home  Plate  Umpire) 

■  Audits  are  performed  on  performance  to  the  Baseline  Management  process  (all  Umpires 
looking  to  see  if  players  are  following  the  process) 

—  Work  Product  and  Configuration  Baselines  are  audited  to  see  if  they  are  correctly 
controlled  (Umpires  and  League) 


Integrity  of  the  process  and  products  are  verified 

CM  Ml’ 
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Configuration  Baseline 
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Configuration  Management 

—  Identify  Configuration  Baselines 

—  Create  Configuration  Baseline 

—  Change  Configuration  Baseline 


Baseball 

—  Innings:  identified  in  Baseball  Rules 

—  First  Inning 

—  ....  Ninth  Inning 


Comments 

—  As  the  Configuration  Baseline  evolves,  the  status  aeeounting  data  is  maintained 
(similar  to  the  evolving  seore  in  baseball). 

—  The  seore  at  the  end  of  eaeh  inning  is  a  snapshot  in  time 


As  the  game  progresses  the  score  (Conf  Baseline)  evolves 


CMMf 
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•  Configuration  Management 

—  Identify  Product  Baseline/TDP 
—  Control  Product  Baseline/TDP 
—  Deliver  Product  Baseline/TDP 


•  Comments 

—  The  game  (components  of  Product  Baseline/TDP)  is  identified  ahead  of  time 

—  The  game  is  conducted  and  statistics  kept  about  performance  (Baseline  Management 
and  Status  Accounting) 

—  The  baselined  product  is  delivered  (final  score).  Winning  depends  on  how 
successful  the  teams  were  in  scoring/developing  and  controlling  good  work  products. 

—  Errors  have  consequences,  some  impact  the  game  more  than  others  (the  game  could 
be  prolonged/stretched  out  impacting  period  of  performance) 


•  Baseball 

—  Identify  schedule  for  a  game 
—  Conduct  game 
—  Complete  9  innings 


As  the  game  progresses  errors  can  be  disastrous  to  success 

© 


CMMI 
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A.  /'Tfc  .4_«  o  \  Space  and  Airborne 

Opponents  (Preventing  Success)  systems 


•  Configuration  Management 

—  Insufficient  Configuration  Mgmt 
—  No  Defined  Process 
—  Poor  Planning 
—  Poor  Execution 
—  Poor  Leadership 
—  Poor  Team  Cohesiveness 
—  Lack  of  Maturity 
—  Lack  of  Training 
—  Lack  of  Sufficient  Resources 

•  Baseball 

-  Opposing  Team 

—  Owners 

—  Poor  Team  Execution 

—  Poor  Team  Leadership 

—  Poor  Team  Cohesiveness 

—  Lack  of  Player  Maturity 
—  Lack  of  Player  Training 

•  Comments 

—  Many  factors  can  hinder  successful  delivery  of  the  Product  Baseline/TDP  on  a 
program 

—  With  insufficient  Configuration  Management,  it  is  difficult  to  successfully  track 
the  evolving  Configuration  Baseline  and  deliver  the  Product  Baseline 

© 


CMM! 
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Results  of  Winning 
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•  Configuration  Management 

—  Ability  to  easily  provide  any  Work 
Produet  Baseline  or  Configuration 
Baseline 

-  Repeat  Customers 
—  New  Customers/Programs 

•  Baseball 

—  Happy  Owners 
—  Loyal  fans 

—  New  fans 

—  Highly  paid  players/endorsement 
offers 

•  Comments 

—  With  successfully  controlled  baselines  and  deliveries,  a  company  has  a 
high  probability  of  obtaining  new  programs  and  repeat  customers. 

CMM! 
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•  Ultimately,  to  win  a  baseball  game,  a  team  must  be  able  to 
successfully  score  points  and  defend  against  their  opponents 

•  Owners  drive  the  success  or  failure  of  both  the  CM  processes 
and  Baseball  teams.  However,  in  the  CM  processes  all 
participants  are  owners  of  the  process,  whereas  only  one  rich 
guy  owns  the  ball  club. 

•  To  be  successful  at  delivering  the  correct  product  to  your 
customer 

—  A  Baseline  Management  process  must  be  defined  and  followed 

—  Work  Product  Baselines  must  be  identified,  controlled,  and  managed 

—  Configuration  Baselines  must  be  established  and  maintained 

—  Product  Baselines/TDPs  created  and  delivered  from  the  controlled 
Baselines 


Summary 


CMMI 
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Acronyms 


CCB 

Configuration  Control  Board 

CM 

Configuration  Management 

CMMI 

Capability  Maturity  Model  Integrated 

CR 

Change  Request 

CRB 

Change  Review  Board 

EO 

Engineering  Order 

ERB 

Engineering  Review  Board 

PCR 

Program  Change  Request 

PRB 

Program  Review  Board 

SCR 

Software  Change  Request 

SPCR 

Software  Problem  Change  Request 

STR 

Software  Trouble  Report 

TDP 

Technical  Data  Package 

Raytheon 

Space  and  Airborne 
Systems 
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System  assurance  is  the  level  of 
confidence  that  the  system  functions 
as  intended  and  is  free  of  exploitable 
vulnerabilities,  either  intentionally  or 
unintentionally  designed  or  inserted 
as  part  of  the  system. 
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Ti  Assurance  Problem 
Space 


■  Large-scale  systems  and  systems  of  systems  represent  a 
complex  supply  chain  integrating 
I  Proprietary  and  open-source  software 
I  Legacy  systems 
I  Hardware 
I  Firmware 


■  These  systems  are  sourced  from  multiple  suppliers  who  employ 
people  from  around  the  world 

■  Most  systems  we  encounter  today  contain  software  elements 
and  most  depend  upon  software  for  a  good  portion  of  their 
functionality 

■  Technologies  to  build  reliable  and  secure  software  are 
inadequate 

I  Our  ability  to  develop  software  has  not  kept  pace  with  hardware 
advances 


I  Can6  construct  complex  software-intensive  systems  for  which  we 
can  anticipate  performance 

Assurance  is  a  full  life  cycle  systems-level  problem 
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are  As  A  Root  Cause 
Problem 


■  System  risk  has  dramatically  increased  due  to  the 
simultaneous  growth  in  software  vulnerabilities  and 
in  threat  opportunities 

■  Risk  management  processes  inadequately  address 
these  threats  and  r  sks 

■  Threats  presented  by  suppliers  of  software  products 
and  services  are  not  adequately  Identified  and 
analyzed 

■  Development  and  acquisition  processes 
inadequately  address  software  security 

■  There  is  a  fundamental  lack  of  both  the  scientific 
understanding  of  software  risks  and  the  capabilities 
to  effectively  diagnose  and  mitigate  in  the  in  a  timely 
manner 


Source:  J.  Jarzombek.  DOD  Software  Assurance 
Initiative:  Mitigating  Risks  Attributable  to  Software. 
DOD  Software  Assurance  Forum,  July  2004. 
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More  Succinctly  . . . 


■  There  is  a  failure  to  assure  correct, 
predictable,  safe,  secure  execution  of 
complex  software  in  distributed 
environments 

■  Inadequate  attention  is  given  to  the  total 
life  cycle  issues,  including  impacts  on  life 
cycle  cost  and  risk  associated  with  the  use 
of  commercial  or  reused  products  and 
components 


Source:  G.  Draper  (ed.),  Top  Software  Engineering  Issues  Within  Department  of  Defense  and 
Defense  Industry.  National  Defense  Industrial  Association,  Arlington,  VA,  August  2006. 
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Systems  Engineering 
Challenge 


Integrating  a  heterogeneous  set  of  globally 
engineered  and  supplied  proprietary,  open- 
source,  and  other  software;  hardware;  and 
firmware;  as  well  as  legacy  systems;  to 
create  well-engineered  integrated, 
interoperable,  and  extendable  systems 
whose  security,  safety,  and  other  risks  are 
acceptable  i  or  at  least  tolerable. 
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lystem  and  Software  Assurance 
CMMI®-Connpliant  Processes 


1 .  Understand  Your 
Business 
Requirements  for 
Assurance 


2.  Look  to  the 
CMMI®for 
Assurance -Related 
Process  Capability 
Expectations 


5.  Measure  Your 
Results  -  Modify 
Processes  as 
Necessary 


4.  Build  or  Refine 
and  Execute  Your 
Assurance 
Processes 


3.  Look  to 
Standards  for 
Assurance 
Process  Detail 
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®-  DEV  Assurance 
Shortfalls 


*  Inconsistent  treatment 
of  safety  and  security 
concerns 

*  Insufficient  assurance 
detail  in  required  and 
expected  components 

V  Specific  goals 

V  Specific  practices 

*  Insufficient  traceability 
to  assurance  source 
standards 
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CMMI®T 
DEV 
Process 
Areas  and 
Assurance 


Source:  CMMI®  for 
Development,  Version 
1.2,  CMU/SEI-2006- 
TR-008,  August  2006 


Name 

Abbr 

Safety 

Security 

Requirements  Management 

REQM 

V 

V 

Project  Planning 

PP 

V 

V 

1  Project  Monitoring  and  Control 

PMC 

V 

Supplier  Agreement  Management 

SAM 

V 

Measurement  and  Analysis 

MA 

V 

Process  and  Product  Quality  Assurance 

PPQA 

Configuration  Management 

CM 

V 

V 

Requirements  Development 

RD 

V 

V 

Technical  Solution 

TS 

V 

V 

Product  Integration 

PI 

V 

V 

Verification 

VER 

Validation 

VAL 

Organizational  Process  Focus 

OPF 

Organizational  Process  Definition  +IPPD 

OPD  +IPPD 

V 

V 

Organizational  Training 

OT 

V 

V 

Integrated  Project  Management  +IPPD 

IPM  +IPPD 

V 

V 

Risk  Management 

RSKM 

V 

V 

Decision  Analysis  and  Resolution 

DAR 

V 

Organizational  Process  Performance 

OPP 

Quantitative  Project  Management 

QPM 

Organizational  Innovation  and  Deployment 

OID 

Causal  Analysis  and  Resolution 

CAR 

V 
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and  Security  Extensions  for 
i  Capability  Maturity  Models  '! 


Take  1 


1.  Ensure  Safety  and  Security  Competency 

2.  Establish  Qualified  Work  Environment 

3.  Ensure  Integrity  of  Safety  and  Security  Information 

4.  Monitor  Operations  and  Report  Incidents 

5.  Ensure  Business  Continuity 

6.  Identify  Safety  and  Security  Risks 

7.  Analyze  and  Prioritize  Risks 

8.  Determine,  Implement,  and  Monitor  Risk  Mitigation 
Plan 

9.  Determine  Regulatory  Requirements,  Laws,  and 
Standards 

10.  Develop  and  Deploy  Safe  and  Secure  Products  and 
Services 

11.  Objectively  Evaluate  Products 

12.  Establish  Safety  and  Security  Assurance  Arguments 

13.  Establish  Independent  Safety  and  Security  Reporting 

14.  Establish  a  Safety  and  Security  Plan 

15.  Select  and  Manage  Suppliers,  Products,  and  Services 

16.  Monitor  and  Control  Activities  and  Products 


Safety  anil  Seeiirity  Kxtt‘nsi<ms 

ftn 

Intejirated  C'apuhility  Matut  it%  Models 


Lindn  Ihi  ahim 
Joe  JiMvomtiek 
Matt  Ashlbrcl 
Ro|;er  I  Jute 
Vnu\  (  roll 
Mim  El  urn 
[.aIJrijycre 
’  <Tirt\Velh 

unci  the  Mc.^mhers  nt'the 
Safety  and  Seeurity  Kxtensions  lYoject  J  eam 


Septemher  2004 


Source:  United  States  Federal  Aviation  Administration,  Safety  and  Security  Extensions  for  Integrated  Capability  Maturity  Models,  September 
2004  r-ittp://www,faa.qov/abG-jt.*office  org/h^ad quarters  or",.  'aio/docurr:ents/rT:ed  i/Safetv-^ndSecuritvizxt-FINAL-web.ijdf) 
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iource  Standards 


Safety 


Security 


Defence  Standard  00-56,  Safety 
Management  Requirements  for 
Defence  Systems,  Ministry  of 
Defence,  United  Kingdom, 
December  1996. 

lEC  61508,  Functional  Safety  of 
electrical/electronic/programmable 
electronic  safety- related  systems. 
International  Electrotechnical 
Commission,  1997. 

Military  Standard  System  Safety 
Program  Requirements,  MIL-STD- 
882C,  United  States  Department  of 
Defense,  January  1993. 

Standard  Practice  for  System 
Safety,  MIL-STD-882D,  United 
States  Department  of  Defense, 
February  2000. 


ISO/IEC  21827,  Systems  Security 
Engineering  Capability  Maturity 
Model®,  SSE-CMM®,  Model 
Description  Document,  Version  3.0, 
June  15,  2003. 

ISO/IEC  15408:1999,  Common 
Criteria  for  Information  Technology 
Security  Evaluation,  Part  3:  Security 
assurance  requirements.  Version  2.1 , 
1999. 

ISO/IEC  17799:2000(E):  Information 
technology  i  Code  of  practice  for 
information  security  management. 
International  Organization  for 
Standardization,  First  edition  2000- 
12-01. 

Risk  Management  Guide  for 
Information  Technology  Systems, 

National  Institute  of  Standards  and 
Technology,  Special  Publication 
800-30,  2001. 


Source:  United  States  Federal  Aviation  Administration,  Safety  and  Security  Extensions  for  Integrated  Capability  Maturity  Models,  September 
2004  r.;ttp://www,faa.qov/abG-jt.*office  org/hr  quarters  of",,  'aio/docurr:ents/iT:ed  i/Safety-^ndSecurity-xt-FINAL-web-odf) 
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Extensions  for  Integrated 
Maturity  Models  T  Take  2 


■  Workshop  on  Assurance  with  CMMI®,  August  7,  2007 
T  Relationships  between  Models  and  Standards 
I  Industry  experiences  in  extending  models  for  assurance 

■  Motorola®  Secure  Software  Development  Model 

■  Lockheed  Martin®  Software  Safety  and  Security  Certification 
Best  Practices 

K  Booz  Allen  Hamilton®  experience  with  multiple  models 

T  Community  of  interest  feedback  on  security  extensions  to 
the  CMMI® 


Security  Model  Harnnonization  Working  Group 

T  Harmonization  of  key  security  capability  maturity  models 
including  but  not  limited  to  the  SSE-CMM  and  the  Motorola 
Secure  Software  Development  Model  (MSSDM) 

T  Prototyping  Assurance  as  a  fFocus  Areao 

T  Assurance  beginning  with  Security  in  Phase  I  adding  Safety 
and  Dependability  in  Phase  II 
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3r-The-Buck  CMMI®-DEV 
anagement  Process  Areas 


$$$ 


RSKM 

\ _ J 


I  Identify,  Evaluate,  Categorize,  and  Prioritize 
Assurance  Risks 

I  Develop  assurance  risk  mitigation  strategies 


$$ 


PP 


I  Determine  a  technical  approach  for  the  project  that 
supports  the  assurance  requirements 

V  Determine  the  level  of  security  required  for  tasks,  work 
products,  hardware,  software,  personnel,  and  work 
environment 


$$ 


PMC 


I  Monitor  significant  changes  in  risk  status 
V  Monitor  the  security  environment 


$$ 


SAM 


I  Evaluate  COTS  products  for  compliance  with 
assurance  requirements 

I  Evaluate  the  trustworthiness  of  the  supplier 


CM  Ml®  for  Development,  Version  1.2,  CMU/SEI-2006-TR-008,  Software  Engineering  Institute,  Carnegie  Mellon  University,  August  2006 
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“1 — I  WOOOO" 


or-The-Buck  CMMI-DEV® 
Jianagement  Process  Areas 


$$ 


Establish  and  maintain  training  capability  to  address 
assurance-related  training  needs 

Provide  training  necessary  to  ensure  the  competency 
of  individuals  required  to  perform  assurance-related 
roles  effectively 


CM  Ml®  for  Development,  Version  1.2,  CMU/SEI-2006-TR-008,  Software  Engineering  Institute,  Carnegie  Mellon  University,  August  2006 
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or-The-Buck  CMMI-DEV® 
l^igrlieering  Process  Areas 


$$$ 


RD 

V 

/ 

I  Identify  customer  expectations  for  assurance 
T  Define  product  assurance  attributes 


$$$ 


TS 

V 

/ 

I  Identify  and  analyze  alternative  solutions  based  on  proposed 
product  architectures  that  address  critical  product  qualities 

I  Ensure  that  the  detailed  design  adheres  to  applicable 
assurance  standards  and  criteria 


$$ 


VER 


T  Select  verification  methods  based  on  their  ability  to 
demonstrate  that  the  work  product  properly  reflects  the 
specified  assurance  requirements 

I  Establish  and  maintain  the  environment  needed  to  support 
validation,  including  test  tools  and  simulations 


$$ 


I  Select  validation  methods  based  on  their  ability  to  demonstrate 
that  customer  expectations  for  assurance  are  satisfied 

I  Establish  and  maintain  the  environment  needed  to  support 
validation,  including  test  tools  and  simulations 


CM  Ml®  for  Development,  Version  1.2,  CMU/SEI-2006-TR-008,  Software  Engineering  Institute,  Carnegie  Mellon  University,  August  2006 
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~  OUl 


r-The-Buck  CMMI-DEV® 

pport  Process  Areas 


$$$ 

CM 

V  J 

Create  a  baseline  that  can  be  changed  only  through 
formal  change  control  procedures 

Perform  reviews  to  ensure  that  changes  have  not 
compromised  the  safety,  security,  or  dependability 


$$ 


Objectively  evaluate  the  work  products  against  the 
applicable  assurance  process  descriptions,  standards, 
and  procedures 


CM  Ml®  for  Development,  Version  1.2,  CMU/SEI-2006-TR-008,  Software  Engineering  Institute,  Carnegie  Mellon  University,  August  2006 
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dance  For  Systems 
^  Assurance  - 1 


■  Systems  Assurance  -  Delivering 
Mission  Success  in  the  Face  of 
Developing  Threats 

V  An  NDIA  guidebook  intended  to 
supplement  the  knowledge  of  systems 
(and  software)  engineers  who  have 
responsibility  for  systems  for  which 
there  are  assurance  concerns 
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““  I  w  I 


Bystem  Assurance  Guidebook  i 
dTo  ISO/IEC/IEEE  15288 


■  Agreement  Processes 

I  Acquisition 
I  Supply 

■  Project  Processes 

I  Project  Planning 
I  Project  Assessment 
I  Project  Control 
I  Decision-making 
I  Risk  Management 
I  Configuration  Management 
I  Information  Management 

■  Assurance  Case  Process 


■  Technical  Processes 

I  Stakeholder  Requirements 
Definition 

I  Requirements  Analysis 
I  Architectural  Design 
I  Implementation 
I  Integration 
V  Verification 
I  Transition 
I  Validation 
I  Operation 
I  Maintenance 
I  Disposal 


Enterprise  Processes 

I  Enterprise  Environment 
Management 

I  Investment  Management 


I 


System  Life  Cycle  Process 
Management 

Resource  Management  [including 
human  resource  training] 

Quality  Management 
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Df  Standards  In  The  Guidebook 


Concept 

Stakeholder  Heeds 

Ei^jljOTe  Coiicepts 

Propose  Idable  Solutiore 

Development 

Refine  System  Requirements 

Production 

Utilization 

Operate  System  to 

Satis^  User  Heeds 

Retirement 

Create  Solutioii  Desorption 

Build  System 

Wlidate 

Produce  Systems 
frispectfi  Test 

Support 

> - 

ProTTide  Sustained 
System  Capability 

Store,  ArchiiTe,  or 

Dispose  ofthe  System 

ISO/IEC  15288 :2002(E) 


IEEE  1220-2005 


Concept 

System 

Definition 

Preliminary 

Desisn 

Detailed 

Desisn 

FAIT* 

Production 

'  Utilization 

Support 

Retirement 

Support 

*  Fabiicatiaii, 

Intfigratiorv  ^ 


Integrated  Defense  Acquisition,  Technology,  &  Logistics  Life  Cycle  Framework 

1  ME  A  MSB  "  M3  C 

_ A _ _ A- 


JCIDS 

Heeds: 

FAA 

FHA 

FSA 


Concept 

Decisicn 


DCR  ICD 


(KR-KPP) 


Concept 

Refinement 


Defense  mua] 
Acquisition 
System 


(EP) 


lU, 


Technology 

Development 

ITR-KPP 


FEP 


(EP) 


SEP'- 
1  CDDd 

-Pre-3ystemfAcquisition  ■ 


EP 


seJp 

lOA 
CDD  1 
- 


Sy stem  Development  & 
Demonstration 
I  HE-KPP 


IOC  FOC  ^ 

A  A\ 


Production 

& 

Deployment 


Systems  /  Systems 

fcitegpdicii  /  Demonstnition 

System  Rinftional  cpp 

- System  A  c  quisition 


W, 


Operations 

& 

Support 


I({A" 


^EP 

LEIP 


Final  ProdcLct] 
Baseline 


(EP) 

SustainmentJ 

Sustainment  ■ 


Initiation 

A  c  quisition/D  e  velopment 

I  mple  mentation/A  ss  essment 

Operations  Maintenance 

Disposition 

1  -  Business  Partner 

1  -  Risk  Assessment 

1  -  Prodict/Cooiponeit  fcispection  ft 

1  -  Change  Control  ft  Auditing 

1  -IhnsitionPlarning 

2  -  DoomeiilEliteipiise  Ardutecture 

2 -Mitial  Security  Baseline  Coitrols 

Acceptance 

2  -  Contiiuous  Monioring 

2  -  Conponent  Disposal 

3  -  Identi^/Specifijc  Applicable  Policies  &  Laws 

3  -  Refinement  of  S  ecurity- Baseline 

2  -  Security  Cantrol  Mteg^tion 

3  -Re-Certification 

3  -  Media  SanitatierL 

4  -  Derrelop  C,L&  A  ObjectiireE 

Corlrot 

3  -User/AdministiatiiTe  Ouidance 

4  -Re -Ac  ere  dilation 

4  -  information  Ardniving 

5  -  &i[farmatian&  Mbaiiatiofn  System  Security- 

4  -  Security  Control  Baseline 

4  -  System  Security  Test  ft  EJvaluaticfi 

5  -kucident  Handing 

Cate  gprisaticn 

5  -  Cost  Ana^is  &  Reporting 

Plan 

6  -  Audiing 

6  -  Process  SpecificaLLotnDeTTelopment 

6  -  Security  Planmiig 

5  -  Security  Certificatiem 

7  -kttiusiin  Detection  ft  Monitor^ 

7  -  Preliminary  Risk  Assessment 

7  -  TJbit/kT[teg7itio(n  Security  Test  ft 

6  -  Statement  of  Residual  Risk 

0  -  Contingency  Planning  (includi^  Continuiyaf 

Evaluation 

7  -Security  Accreditation 

Operatiens  Plan  [COOP]) 

KIST  Information  Security  and  the  System  Development  Life  Cycle 
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dance  For  Systems 
*  Assurance  -  2 


■  state  of  the  Art  Report  on  Software 
Security  Assurance 

V  An  lATAC/DACS  report  identifying  and 
describing  the  current  state  of  the  art  in 
software  security  assurance,  including  trends 
in: 

■  Techniques  for  the  production  of  secure  software 

■  Technologies  that  exist  or  are  emerging  to  address 
the  software  security  challenge 

■  Current  activities  and  organizations  in  government, 
industry,  and  academia,  in  the  U.S.  and  abroad, 
that  are  devoted  to  systematic  improvement  of 
software  security 

■  Research  trends  worldwide  that  might  improve  the 
state  of  the  art  for  software  security 
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dance  For  Systems 
*  Assurance  -  3 


■  Secure  Software  Assurance:  A 
Guide  to  the  Common  Body  of 
Knowledge  to  Produce,  Acquire, 
and  Sustain  Secure  Software 

T  A  DHS  guidebook  intended  as  a 
framework  to  identify  workforce  needs 
for  competencies  and  leverage 
standards  and  best  practices  to  guide 
software- related  curriculum 
development 
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■  Security  in  the  Software  Life  Cycie: 
Making  Software  Deveiopment 
Processes  -  and  the  Software  Produced 
by  Them  -  More  Secure 

V  An  DHS  report  providing  a  compendium  of 
methodologies,  life  cycle  process  models, 
sound  practices,  and  supporting  technologies 
that  would,  if  adhered  to,  increase  software 
security 
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■  Software  Assurance  in 
Acquisition:  Mitigating  Risks  to 
the  Enterprise 

V  A  DHS  report  intended  to  provide 
guidance  on  enhancing  supply  chain 
management  through  improved  risk 
mitigation  and  contracting  for  secure 
software 
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■  ISO/IECSC22  T  OWG: 
Vulnerabilities  (OWGV) 

T  Project  22.24772:  Guidance  for 
Avoiding  Vuinerabiiities  through 
Language  Seiection  and  Use 

■  Technical  Report 

■  Comparative  guidance  spanning  multiple 
programming  languages 

■  Goal:  Avoidance  of  programming  errors 
that  lead  to  vulnerabilities 
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■  ISO/I  EC  SC  27  IT  Security  Techniques 

T  ISO/IEC  15408,  Common  Criteria  for  IT  Security 
Evaluation 

T  ISO/IEC  15443,  FRITSA 

■  Part  1 :  A  framework  for  IT  security  assurance 

■  Part  2:  Assurance  methods 

■  Part  3:  Analysis  of  assurance  methods 

T  ISO/IEC  DTR  19791 ,  Assessment  of  Operational 
Systems 

T  ISO/IEC  21827,  System  Security  Engineering 
Capability  Maturity  Model  (SSE  CMM)  revision 

T  ISO/IEC  27000  series  T  Information  Security 
Management  System  (ISMS) 


7th  Annual  CMMI  Technology  Conference,  15  November 2007,  Tracks,  1015 


26 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  m 

and^panded  Features 


ition  In  Support  Of  Assurance  i 
Functional  Safety 


■  I  EC  SC  65A,  Functional  Safety 


lEC  61508,  Functional  Safety  Of  Electrical/ 
Electronic/Programmable  Electronic  Safety-related 
Systems  (7  parts) 

■  Part  1 :  General  requirements 

■  Part  2:  Requirements  for 
electrical/electronic/programmable  electronic  safety- 
related  systems 

■  Part  3:  Software  requirements 

■  Part  4:  Definitions  and  abbreviations 

■  Part  5:  Examples  of  methods  for  the  determination  of 
safety  integrity  levels 

■  Part  6:  Guidelines  on  the  application  of  lEC  61508-2  and 
lEC  61508-3 

■  Part  7:  Overview  of  techniques  and  measures 

Risk-based  approach  for  determining  the  required 
performance  of  safety-related  systems 
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■  lEC  60300  Series,  Dependability 
Management 

■  lEC  61713,  Software  dependability  through 
the  software  life-cycle  processes- 
Application  guide 

■  lEC  60812,  Analysis  techniques  for  system 
reliability  -  Procedure  for  failure  mode  and 
effects  analysis  (FMEA) 

■  lEC  61025,  Fault  tree  analysis  (FT A) 
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■  FIPS  Publication  199,  Standards  for  Security  Categorization  of 
Federal  Information  and  Information  System 

■  FIPS  Publication  200,  Minimum  Security  Requirements  for  Federal 
Information  and  Federal  Information  Systems 

■  NIST  Special  Publication  800-30,  Revision  1,  Risk  Assessment 
Guideline) 

■  NIST  Special  Publication  800-37,  Guide  for  the  Security  Certification 
and  Accreditation  of  Federal  Information  Systems 

■  NIST  Special  Publication  800-39,  NIST  Risk  Management 
Framework 

■  NIST  Special  Publication  800-53  Revision  1,  Recommended 
Security  Controls  for  Federal  Information  Systems 

■  NIST  Special  Publication  800-53A,  Guide  for  Assessing  the  Security 
Controls  in  Federal  Information  Systems 

■  NIST  Special  Publication  800-59,  Guide  for  Identifying  an 
Information  System  as  a  National  Security  System 

■  NIST  Special  Publication  800-60,  Guide  for  Maming  Types  of 
Information  and  Information  Systems  to  Security  Categories 


^Federal  Information  Security  Management  Act  of  2002 
Source:  httD://csrc.nistaov/sec-cert/ca-DrohPhases.html 
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ISO/IEC/IEEE  15026,  System  and  Software  Assurance 


24748:  Guide  to  Life  Cycie  Management 


Other 
standards 
providing 
detaiis  of 
seiected  SW 
processes 


Source:  J.  Moore,  SC7 
Liaison  Report,  IEEE 
Software  and  Systems 
Engineering  Standards 
Committee,  Executive 
Committee  Winter 
Plenary  Meeting, 
February  2007. 


Revised  12207: 

Life  cycle 
processes  for 
SW 


Revised 

15289: 

Document¬ 

ation 


V/ 


Revised 

16326: 

Project 

^Mgmt^ 


Revised 

15939: 

Measure¬ 

ment 


Revised 

16085: 

Risk 

Mgmt 


Revised  15288: 

Life  cycle 
processes  for 
systems 


Interoperation 


other 
standards 
providing 
detaiis  of 
seiected 
system 
processes 


15026: 
Additional 
practices  for 
higher 
assurance 
systems 


V/ 


Common  vocabulary,  process  architecture,  and  process  description  conventions 
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1 5288  And  1 2207 
fe  Cycle  Processes 


Source:  I  SO/I  EC 
CD  15026/4  IEEE 
P15026/CD1, 
Systems  and 
software 
engineering  — 
Systems  and 
software  assurance 


Organization 


Project-Enabling 

Processes 


Life  Cycle  Model 
Management 


Infrastructure 

Management 


Project  Portfolio^ 
Management 
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Management 


Quality 

Management 
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Processes 


Supply 


Acquisition 
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Project 


Project  Mgmt 
Processes 


Project  Planning 


Project  Assess¬ 
ment  &  Control 


Project  Support 
Processes 
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Management 


Risk  Management 


Configuration 

Management 
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Management 


Measurement 


Engineering 


Technical 

Processes 


Stakeholder 
Requirements  Defn 


SW  Implementation 
Processes 


Requirements 

Analysis 


Architectural 

Design 


Implementation 


Integration 


Verification 


Transition 


Validation 


Operation 


Maintenance 


Disposal 


0) 
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SW  Detailed  Design 


SW  Requirements 
Analysis 


SW  Architectural 
Design 


SW  Construction 


SW  Integration 


SW  Qualification 
Testing 


SW  Support 
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SW  Documentation 
Management 


SW  Configuration 
Management 


SW  Quality 
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SW  Verification 


SW  Validation 


SW  Review 
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SW  Problem 
Resolution 
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Processes 


Domain 

Engineering 


Reuse  Asset 
Management 


Reuse  Program 
Management 
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Of  The  Assurance  Case 


Set  of  structured  assurance 
claims,  supported  by  evidence 
and  reasoning,  that  demonstrates 
how  assurance  needs  have  been 
satisfied. 

T  Shows  compliance  with  assurance 
objectives 

T  Provides  an  argument  for  the 
safety  and  security  of  the  product 
or  service. 

T  Built,  collected,  and  maintained 
throughout  the  life  cycle 

V  Derived  from  multiple  sources 


■  Sub-parts 

T  A  high  level  summary 

T  Justification  that  product  or 

service  is  acceptably  safe,  secure, 
or  dependable 

V  Rationale  for  claiming  a  specified 
level  of  safety  and  security 

T  Conformance  with  relevant 
standards  and  regulatory 
requirements 

T  The  configuration  baseline 

V  Identified  hazards  and  threats  and 
residual  risk  of  each  hazard  and 
threat 


System,  Software,  or  Work  Product 


Make  the  case  for  adecjuate  quality/  assurance  of  the 


justify  belief  in 


Quality  /  Assurance  Case 


Claims 


Arguments 


Evidence 


supports 

Quality  /  Assurance 
Factor 


is  developed  for 

O 


Quality  /  Assurance 
Subfactor 


Operational  and  support 
assumptions 


Attributes 


□ 

Clear 

□ 

Consistent 

□ 

Complete 

□ 

Comprehensible 

□ 

Defensible 

□ 

Bounded 

□ 

Addresses  all 
life  cycle  stages 
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■  ADD  is  a  methodology  used  to  define  a  system 
architecture  that  bases  the  decomposition  process  on  the 
quality  attributes  the  system  (software)  has  to  fulfill. 

■  The  architectural  design  using  the  ADD  methodology  can 
begin  when  the  architectural  drivers  are  known  with 
some  level  of  confidence. 


CNJ 


In  ADD  Tactics  and  Architectural  patterns  are  selected  to 
satisfy  a  set  of  quality  attributes  within  a  critical  scenario 
that  provides  context  for  those  quality  attributes 
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■  Creating  the  business  case  for  the  system 

■  Understanding  and  documenting  the  requirements 

■  Leveraging  Quaiity  Attribute  Scenarios 

■  Creating  or  seiecting  the  architecture 

■  Documenting  and  communicating  the  architecture 

■  Anaiyzing  or  evaiuating  the  architecture 

■  Impiementing  the  system  based  on  the  architecture 

■  Ensuring  that  the  impiementation  conforms  to  architecture 
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■  Prioritized  Business  Goais 

■  Business  goals  associated  with  the 
project  are  elicited  from  selected  project 
stakeholders 

■  Business  goals  are  prioritized  for 
stakeholders  to  guide  architectural 
tradeoffs 

■  Exampie  of  prioritized  business 
goals: 

■  Lower  commissioning  costs  by  xx% 

■  Ensure  system  is  available  99.9% 

■  Maintain  current  system  performance 

■  etc 
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00 


Architectural  drivers  (quality  attribute 
scenarios)  include  the  combination  of 
functional  and  quality  requirements  that 
shape  the  architecture: 

■  Define  unique  functions  (as  architectural 
Functional  Requirements)  of  modules  in  the 
system 

■  Select  associated  Non-functional 
Requirements 

■  Quality  attribute  scenarios  provide  the 
functional  context  under  which  Non 
Functional  Requirements  are  defined 

■  Architectural  patterns  that  satisfy  the  critical 
scenarios  are  then  selected 
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SP1.1  Elicit  needs 


SP  1.2  Develop  the  customer  (architectural)  requirements 


Use  Case 

The  operator  runs  a  sequence 
of  complex  applications 


Customer  (Architectural) 

Requirements 

Includes  Functional  and 
Non-functional  requirements 


The  system  shall  allow  the  operator  to  run 
the  state  estimator  application 

The  system  shall  allow  the 
operator  to  run  sensitivity  analyses 

The  system  shall  allow  the 
operator  to  run  the  PS  model 

The  system  shall  allow  the  operator  to  run 
a  sequence  of  applications  in  an 
“industry  acceptable”  time 
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SP1.1  Elicit  needs 


SP  1.2  Develop  the  customer  (architectural)  requirements 


Quality  Attribute  Customer-reiated 

System  Quality  Non  Functional  Requirements 

Associated/derived  from 
Quality  Attribute 


The  source  code  for  the  system  shall 
not  be  modified  for  any  customer 
implementation 


Commissionability 


The  complete  system  installation  shall  be 
completed  in  an  “acceptable”  time  period 
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SP  2.1  Establish  product  and  product  component  requirements 
SP  2.2  Allocate  product  component  requirements 
SP  2.3  Identify  interface  requirements 


Customer  Requirements 

Includes  Functional  and 
Non-functional  requirements 


Product  Architectural  Requirements 

Testable  and  measurable 
set  of  requirements 


The  system  shall  allow  the  operator 
to  run  the  state  estimator  application 


The  system  shall  allow  the  operator  to  run 
the  state  estimator  application  in  xx  seconds 


The  system  shall  allow  the  operator  _ ^  The  system  shall  allow  the  operator  to 

to  run  sensitivity  analyses  run  sensitivity  analyses  in  yy  seconds  per  run 


The  system  shall  allow  the  _ ^  The  system  shall  allow  the  operator 

operator  to  run  the  PS  model  to  run  the  PS  model  in  xy  seconds 


The  system  shall  allow  the  operator 
to  run  a  sequence  of  applications  in 
an  “industry  acceptable”  time 


The  system  shall  allow  the  operator  to  run 
a  sequence  of  applications  in  yz  seconds 
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Attribute  Scenarios 


Environment  Response 

Measure 


■  Encapsulate  a  set  of  architectural  functional  and  non¬ 
functional  requirements  that  uniquely  define  the  system  being 
architected 


■  Are  described  by  a  set  of  detailed  architectural  product 
requirements 


■  Can  incorporate  of  one  or  more  Use  Cases 
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SP  3.1  Establish  operational  concepts  and  scenarios 
SP  3.2  Establish  a  definition  of  required  functionality 
SP  3.3  Analyze  requirements 
SP  3.4  Analyze  requirements  to  achieve  balance 
SP  3.5  Validate  requirements 


Quality  Attribute  Scenario 

Sequence  Diagram 


Detailed  Architectural 
Non  Functional  Requirements 

Placed  in  context  of  Critical  Scenario 


The  time  duration  of  sequence 
calculations  shall  be  less  than 
XX  seconds  under  normal 
loading  conditions 

The  performance  of  running  the 
numerical  application  sequence 
shall  be  such  that  it  will  not  exceed 
specified  bounds  of  memory  and 
CPU  load  capabilities 
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SP  1.1  Obtain  an  understanding  of  requirements 

SP  1.2  Obtain  commitment  to  requirements 

SP1.3  Manage  requirements  changes 

SP  1.4  Maintain  bi-directional  traceability  of  requirements 

SP  1.5  Identify  inconsistencies  between  project  work  and  requirements 


cmp  NM  Non-functional  Requirements 


Build  Time  NFRs 

U  1  1  1  J 

+  Commissionability 
+  Interoperability 
+  Maintainability 
+  Portability 
+  Testability 

Cross-Cutting  NFRs 

1  1  1  U 

+  Affordability 
+  Data  Integrity 
+  Marketability 
+  Security 

Run  Time  NFRs 

1 1 1 1  j 

+  Availability 

+  Performance 

+  Scalability 
+  Usability 

Understanding  and  commitment  to  requirements 
among  stakeholders  carried  out  through  meetings 


Functional  and  Non  Functional  requirements 
Stored,  managed,  and  maintained  in 
Enterprise  Architect  and  Requisite  Pro  tools 
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The  practices  of  the  RD  process  area  greatly 
contribute  to  defining  the  functional  and  non¬ 
functional  architectural  requirements  that  form  the 
basis  for  ADD 


■  Organization  business  objectives  are  essential  to 
establish  priorities  that  drive  the  development  of  the 
architecture 


■  Quality  attribute  scenarios  provide  context  to  non¬ 
functional  requirements 

■  To  implement  quality  attribute  scenarios,  specific 
tactics  identified  in  ADD  provide  architectural 
patterns 
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Michael  T.  Kutch,  Jr.  Mike  Knox 

SPAWAR  Systems  Center  Charleston  (SSC-C)  Technical  Software  Services,  Inc. 

Head,  Intelligence  &  Information  Warfare  Systems  Director,  Implementation  and  Support 
Engineering  Department  g^,  Authorized  instructor 

Nationai  Competency  Lead  for  I/A  5.8 

Deputy  National  Competency  Lead  for  ISR/IO  5.6 


7^*^  Annual  CMMI  Technology  Conference  and  Users  Group 

November  12-15,  2007 
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Connecting  the  Warfighter 


Mission-  We  enable  knowledge  superiority 
to  Naval  and  Joint  Warfighters  through  the 
development,  acquisition,  and  life-cycle 
support  of  effective,  integrated  C4ISR 
Information 
Technology, 
and  Space 
capabilities. 

Vision- 
Fully  Netted 
in  Three 

We  are  the 
Principal  C4I 
Acquisition 
Engineering  & 

Integration 
Center  on  the 
East  Coast 
&  Principal 
C4ISR  ISEA  for 

the  Navy 


What  We  Do 


Speed  to 
Capabilitv 


LtGHTSpeet} 


MWR-  MobileNet 


Leveraging 

Technology 


□u 


Body  Worn 
Variant 


rH  NETCOP-Network  Common 
^*1  Operating  Picture 
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Rapid 

Prototyping _ _ ^ 
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> Vision  and  Strategy 

>  Elements  of  Implementation 

> Process  Asset  Library 
>Tools 

>ePlan  Builder  and  eWBS 

>  Organizational  Measurement  Repository 

>Training 

>  Training  Architecture 

>  Courses 

>  Resuits 
>Going  Forward 
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Process  Improvement  and 
Systems  Engineering  Strategy  -  2003 


AVision 

I  Develop  and  maintain  a  World  Class  Systems  Engineering  Organization 

AApproach 

I  Achieve  Command-wide  operational  consistency 
T  Based  on  ISO  15288  T  systems  engineering 
I  Based  on  ISO  12207  i  software  engineering 
T  Measure  using  best  practices  of  CMMI® 

ACoals 

T  CMMI  Maturity  Level  2  by  April,  2005 
T  CMMI  Maturity  Level  3  by  April,  2007 


Both  Goals  attained  on  schedule 
SPAWAR  Systems  Center  to  Achieve  ML2  and  MLS 
New  Goal:  Maturity  Level  4  by  2010 
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Cutting  corners, 
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untrained 


Rigorous  processes, 
Skiiied  resources 
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CRITICAL  SUCCESS  FACTORS  FOR  SE  REVITALIZATION 

Command-wide  Policy 
(Create  vision  that  is  urgent) 

Assign  Responsibilities 
(Strong  Change  Agents  are  essential) 

Strategy  and  Plan  (Include 
knowledge  of  why  change  is 
necessary  and  benefits) 

Provide  Training 

Senior  Management  Support 

Build  Central  Repository 

Provide  Resources  and  Funding 
(New  Organizational  Structure 
Usually  Needed) 

Measure  and  Communicate  Progress 
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Elements  of  SSC-C  SE  Revitalization 
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SSC-C  SE  Instruction 


SSC-C  SE 
Process  Manual 


SSC-C  SW-Dev 
Process  Manual 


SSC-C  SW-Maint 
Process  Manual 


EPO  Website 


Underway 

Completed/Ongoing 


Policy  /  Guidance 

■ 

Training  /  Education 

■ 

Assessment  &  Support 

I 

Intro  to  PI  WBT 

CMMI®  Level  2 

SE  101  WBT 

CMMI®  Level  3 

SE  Fundamentals 

CMMI®  Level  4/5 

SE  for  Managers 

Project  Reviews 

Project  &  Process 

Balanced  Scorecard 

Workshop 

Intro  to  Software  Engr. 

Lean  Six  Sigma 

Integrated  Product 

Architecture  Dev.  WBT 

Teams 

Certification/Degrees 

i-  IT  Tools 
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External 

Liaison 


Mike  Kutch 
SE/CMMI  Champign- 


Vision 


Bruce  Carter 
ir.  Engr.  Operations 


Management 
Steering  Group 
(MSG) 


Strategy 


AD 


RISK' 


Engineering 
Proces'^  Offirp 

(E 


Tacticai  impiementation 


PM  IPT 


Enterprise 
Process  Group 
(Ent  PG) 


Lsr 


n 

"PPQA 
IPT 


CM  IPT 


Tecinn 

IPT 
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ASupports  the  Director  of  Engineering  Operations 
AOeveloped  Policies 

I  Policy  for  each  CMMI  Level  2,  3,  4,  &  5  Process  Area 


Engineering 
Process  Office 
(EPO) 


AOeveloped  Standard  Process  Manuals 

T  Top  Level 

A  Systems  Engineering 
A  Software  Development 
A  Software  Maintenance 


I  Supporting  Processes 

A  Process  Manual  for  each  CMMI  Level  2,  3,  4,  &  5  Process  Areas 
A  Additional  process  documentation  as  needed  i  Reviews,  Tailoring,  etc 

AOevelop  plan  templates 

ACoach  and  mentor  selected  projects 

ABuild  tools 


AOevelop  and  deliver  training 
APerform  interim  assessments 
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Recognized  early  need  for  central  repository 
for  Organizational  Process  Assets 


Templaies 


OPA 


Stamlaird 

ProceasG-s 

Tailoring 

Criteria 

*  PoEkias 

Project  Life 
Cycles 

Measurement 

Repository 

•  Polky 

•  Procfl-as 
Manuals 

*SOPs 

*  Waterfall 

Library  of 
Process-Rfllated 
Dccumenlation 

*  Pro>act  Data 

*€anipie 
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•  Incremanlal 
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•Cost 
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Incomplete 


EPO  website  provides  access  to  aii 
SC-C’s  organizationai  process  assets 


Approximately  100  pages  of  content;  over  1000  documents  available 


sm/AR 

"T 

S^stenis  Center 
ChaHeston 


SSC-Charleston  Engineering  Process  Office 


]  Search 


EPO  Home  |  ePlan  Builder  |  WBT  Courses  |  eWBS  |  Contact  EPO  |  CorpWeb 


Navigation 

Getting  Started  EPO  Home 


Calendar 

SSC-C  Standard 
Processes 


Welcome  to  the  SPAWAR  System  Center  -  Charleston's  Engineering  Process  Office  (EPO)  Homepage.  This 
site  is  the  repository  for  a  wealth  of  systems  engineering,  software  engineering,  and  process 
improvement  information  to  aid  our  vision  in  becoming  a  world-class  systems  engineering  organization. 


Process  Areas 

Projects 

Process 

Improvement 

Teams 

Organizational 

Measurement 

Repository 

Training 

Innovation  Program 
References 


Comments 

Please  direct  comments 
about  or  problems  with 
this  site  to  the  EPO 
Webmaster. 


The  site  contains  the  SSC-Charleston  Organizational  Process  Assets,  including  the  organization's  set  of 
standard  engineering  processes  and  procedures,  tools,  sample  documents,  templates,  and  project 
guidelines.  The  measurement  repository  of  project  and  process  measures  is  also  accessible 

The  site  also  contains  information  about  the  Capability  Maturity  Model  for  Integration  (CMMI®)  and  SSC- 
Charleston's  commitment  to  process  improvement.  The  CMMI®  is  used  to  benchmark  and  measure  our 
process  improvement  progress  against  industry  best  practices. 

Background 

SSC-C  is  committed  to  process  improvement  and  has  been  actively  pursuing  process  improvement 
since  1998.  SSC-C  is  implementing  the  Capability  Maturity  Model  for  Integration  (CMMI®).  The  IDEAL® 
model  is  being  used  to  implement  process  improvement. 

•  SSC-C's  commitment  to  process  improvement  and  policy  regarding  it  were  re-affirmed  in  a  SSC- 
C  command-wide  Process  Improvement  Policy  dated  11  December  2003. 

•  Navy  Endorses  CMMI  as  the  Standard  Process  Improvement  Model 

•  ASN  RDA  Software  Process  Improvement  Initiative 

The  information  below  describes  what  will  be  found  under  each  major  section  of  the  site. 


Upcommg  Events 

10/15/2007 

Architecting  with 

DODAF 

10/22/2007 
10th  Annual: 

Systems 

Engineering 

ConFerence 

11/12/2007 
7th  Annual  CHMI 

Technology 
ConFerence  and 

User  Group 

more  »■ 


Latest  Additions 

2t>OS  Innovation 

Program 
Apphcatian  & 
Guidelines  NEW 

CMMI^  Maturity 
Level  4  Traming 

BrleF 

March  a007 
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spnmR 

SSC-Charfeston  Engineering  Process  Office 

Center 

Chariestan 

I  I 

Search  |  " 

EPO  Home  |  ePian  Builder  |  WBT  Courses  | 

eWBS 

Contact  EPO  |  CorpWeb  | 

Navigation 

Getting  Started 

Calendar 

SSC-C  Standard 
Processes 

Process  Areas 

PrajeHct  Planning 

iPP] 

Proiwut  MonetQnng 
&  Control  CPMC) 

Con  Fig  u  ration 
Management  (CM) 

Process  and 
Product  QualHq^ 
Assurance  (PPQA) 

Regubrem^nts 

Management 

(REQM] 

Measurement  & 
Analysis  (MA) 

Suppler  Agreement 
Management  (SAM) 

Reg  ulrem  ents 
Development  (RD) 

Technical  Solution 
ITS) 


Project  Monitoring  &  Control  (PMC) 


Project  Monitoring  and  Control  (PMC)  is  a  Level  2  (Managed)  Process  Area.  The  purpose  of  PMC  is  to 
provide  an  understanding  of  the  project's  progress  so  that  appropriate  corrective  actions  can  be  taken 
when  the  project's  performance  deviates  significantly  from  the  plan. 

Poiicy  Document 

•  SSC-C  Projenzt  Monitoring  and  Control  Policy 

Process  Manual 

•  SSC-C  Project  Monitoring  and  Control  Process  Manual 


Related  Pracess 
Areas 

Proledb  Pliannrn-q 

fPPl 

Measurement  & 

Analysis  (MA) 


SOPS 

•  In  Process  Review  SOP 

•  Project  Management  Review  SOP 

•  Meeting  SOP 

Sample  Documents 

•  IBFTC  PMC  Plan 

•  CICS  Project  Management  Plan  [PMP) 

•  Towed  Array  Earned  Value  Plan 

Templates 

•  PMP  Plan 


Each  CMMI  process  area 
has  a  standard  page  with 
links  to  policy,  process 
manuai,  SOPs, 
Sample/Project  documents, 
and  other  resources 
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]  ^^earch^ 
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Navigation 


Getting  Started 

Calendar 

SSC-C  Standard 
Processes 

Process  Areas 

Projects 

SCAMPI  Appraised 

Projects 

Self  Assets sed 
Projects 


Process 

Improvement 

Teams 

Organizational 

Measurement 

Repository 


Standard  CMMI  Appraisal  Method  for  Process  Improvement 
(SCAMPI)  Appraised  Projects 

SSC-C  SCAMPI  Appraisal  Summary 

SPAWAR  Systems  Center  -  Charleston  -  Maturity  Level  3 

•  Sponsor:  Mike  Kutch 

•  Projects  Appraised:  AP,  CICS,  IBFTC,  JTWS,  SCN,  VIDS,  NAVMACS  II,  SSES,  Towed  Array 

•  Appraised  27  April  2007 

Integrated  Battle  Force  Training  Center  (IBFTC)  -  Maturity  Level  3 

•  Program  Manager:  Lexine  Langley 

•  Code  856 

•  Appraised  26  January  2007 

Visual  Information  Display  System  (VIDS)  -  Maturity  Level  3 

•  Program  Manager:  Steve  Whitbeck 

•  Code  663 

•  Appraised  15  December  2006 


Each  appraised  project 
has  a  page  and  is 
expected  to  share  good 
examples  of  plans  and 
documents 


Training 

Innovation  Program 
References 


Naval  Modular  Automated  Communications  System  n  (NAVMACS  n)  -  Maturity  Level  3 

•  Program  Manager:  John  Dyar 

•  Code  523 

•  Appraised  15  December  2006 


Comments 


Shipbuilding  and  Conversion,  Navy  (SCN)  -  Maturity  Level  3 
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ePIan  Builder  Tool 


Space  and 
'  IStaval  Warfare 
Systems  Center 
Charleston 

■L'  --  ^ 

'^ePI^h  Builder  41 

T  E  lectfonic  CMffI®  Co  mpNa  nt  Docu  mentation  App  lication 

Save  Qull 

by  tfw-  Op^fiom  -  MicnA&i  Kotch 

ePIan  Builder  tool 


I 


T 

T 


An  interactive,  web-based  application  that  leads  the  user  through 
a  structured  interview  process  (like  TurboTax®)  to  generate  a 
CMMI®-compliant  plan 

Includes  standard,  consistent  text 

Generates  an  initial  project-specific  document 

A  Project  Management  Plan  (with  Work  Breakdown  Structure) 

AConfiguration  Management  Plan 

A  Process  and  Product  Quality  Assurance  Plan 

APequirements  Management  Plan 

AMeasurement  and  Analysis  Plan 

ASupplier  Agreement  Management  Plan  (by  end  of  2007) 

ASystems  Engineering  Plan  (DoD  SEP  Format)  IIech^ft 
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Set‘jp 


Build 
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PMP 


Tailor  each  role 
from  pre-defined 
list  of  tasks  and/or 
add  custom  tasks 


ORGANIZATION 

Organization 

Organization 

Chart 

Program 
Manager  Tasks 

Project  Leader 
Tasks 

Systems 

Engineering 

Tasks 

Security 

Engineering 

Tasks 

Software 

Engineering 

Tasks 

Test 

Engineering 

Tasks 

Configuration 
Manager  Tasks 

Quality 


Project  Leader  Tasks 

The  Project  Leader  is  responsible  for  establishing  and  maintaining  the  project  plan. 

Please  identify  the  specific  responsibilities  of  the  Project  Leader. 


w 

p 

p 

p 

1^ 

p 

p 

p 

p 


Coordinates  all  activities  of  the  prime  contractor  and  subcontractors 
Assigns  specific  responsibilities  to  subcontractors  [PP  GP  2.4] 

Discusses  technical  issues  from  the  Government  with  subcontractors 
Discusses  technical  issues  from  the  subcontractors  with  the  Government 
Manages  the  project  cost  and  schedule  [PMC  1.1]  < - 


Resolves  any  inconsistencies  in  the  reguirements  [PMC  2.2] 
Mitigates  project  risks  [PMC  1.3] 

Manage  and  resolve  corrective  actions  [PMC  2.2]  [PMC  2.3] 

Provides  prime  contractor  and  subcontractor  work  products  and 
deliverables  to  the  Government 


Note  mapping 
to  CMMI® 
generic  and 
specific 
practices 


Please  enter  any  additional  specific  responsibilities  of  the  Project  Leader. 
Task 
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Risk  Identification  in  PMP 


Risks 


This  page  allows  you  to  enter  a  list  of  known  or  expected  risks.  The  severity  of  the  risks 
and  the  mitigation  approach  for  each  should  be  identified.  Please  use  the  table  below  to 
identify  the  major  risks  associated  with  the  project. 

^  Click  for  more  information  about  risks 


Risk  Category 

jschedule 

Risk  Category 

I  Quality  t| 


Risk  Category 


Technics 


Impact/Concern 

Level 

Mitigation  Approach 

Products  are  required  by  ^ 
the  customer  by  10/1/06  ^ 

1  |High 

Be  prepared  to  provide  draft 
materials  if  development  of 

Impact/Concern 

Level 

Mitigation  Approach 

Will  products  be  ready  for  ^ 
10/15/06  in  a  condition  ^ 

1  1  Medium  t| 

Provide  technical  data  to  contractor 
in  accordance  with  schedule  with 

Impact/Concern 

Level 

Mitigation  Approach 

Ability  to  getteh  ^ 

technical  ata  from  the  ^ 

1  |High 

Interact  directly  with  the  satellite  *  j 

manufacturer  to  obtain  the  technical  ^  j 

Aua  hkjtn  \lvtm  O 


PMP  may  also  reference  a  more  comprehensive  Risk  Management  Plan 
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Cost, 

Schedule,  and 
Process 
Performance 
are  standard 
categories  of 
measures 


Cost  is  a  measure  within  the  Financial  Performance  category  that  measures  the  cost  for 
activities^  events,  and  products.  The  measure  provides  an  easy-to  understand  view  of 
the  budget.  Comparison  of  planned  and  actual  cost  data  provides  insight  into  significant 
and  repetitive  cost  changes  at  the  activity  level. 

While  more  detailed  cost  information  provides  more  insight  into  the  project's  total  cost, 
until  the  project  personnel  have  achieved  a  certain  level  of  proficiency  in  estimating 
costs,  it  is  recommended  that  the  cost  data  should  be  captured  at  a  level  commensurate 
with  this  level  of  experience. 

Collection  and  Storage 

Identify  the  level  of  detail  for  capturing  cost  data 
I  Project  Level 


Collection, 
Storage,  and 
Analysis  is 
defined  for 
each  Project 
measure 


Please  select  how  the  Project  Leader  will  report  contract  costs  from  the  list  below.  If  the 
Project  Leader  is  not  responsible  for  managing  contracts,  select  "Project". 

I  Project  U 

Identify  who  will  provide  the  actual  cost  data; 

[Project  Leader  >| 

Identify  the  tool  to  be  used  to  collect  cost  data; 

|bSA  and  PMACS  V 

Identify  how  often  the  actual  cost  data  will  be  collected; 

I  Monthly 

Analysis  Procedures 

Identify  how  often  the  cost  data  will  be  analyzed; 

I  Monthly  ^  I 

Identify  the  cost  alert  threshold; 

1 95% 
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SEP 


NAS  Pensacola 
□  SP  Survey 


Sf  PROJECT  SETUP 

Ef  DOCUMENT 
“  SETUP 

^  PROGRAM 

^  INTRODUCTION 

rf  ACQUISITION 
HISTORY 


Previous  Life- 
Cycle  Phases 

Next  Life-Cycle 
Phase 


rf  SYSTEM 

CAPABILITIES 

S  SE 

ORGANIZATIONAL 

INTEGRATION 

g#  SYSTEM 
ENGINEERING 
PROCESS 

Sf  INTEGRATION 

INTEGRATED 
MASTER  PLAN 


SEP  format  follows  the  DoD  SEP  Preparation  Guide 


Next  Life-Cycie  Phase 

The  SEP  requires  that  the  program's  acquisition  history  and  life-cycle  phase  0  be 

discussed,  describing  the  top-level,  technical  process  used  in  each  life-cycle  phase,  This 
Next  Life-Cycle  Phase  section  should  give  an  overview  of  the  next  planned  life-cycle 
phase  as  well  as  summarize  the  process  activities  that  are  expected  to  be  finished  during 
the  next  life-cycle  phase, 

^lease  enter  text  discussing  the  Next  Life-Cycle  Phase  of  the  program, 

This  description  should  give  an  overview  of  the  planned  SE  process  and  should  have 
more  detail  than  the  historical  life-cycle  processes  completed.  It  should  include  how  the 
technical  process  will  be  integrated  into  the  life-cycle  model  and  summarize  the  process 
activities  that  are  expected  to  be  finished  during  the  next  life-cycle  phase, 

Life-Cycle  Phases  (in  hierarchical  order); 


1, 

2, 

3, 

4, 

5, 


Concept  Refinement 
Technology  Development 
System  Development  and  Demonstration 
Production  and  Deployment 
Operations  and  Support 


5how  Hidden  Text 
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SEP 
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DSP  Survey 

PROJECT  SETUP 

DOCUMENT 

SETUP 

PROGRAM 

INTRODUCTION 

ACQUISITION 

HISTORY 

SYSTEM 

CAPABILITIES 

System 

Capabilities 

Certification 
Requirements 

Design 

Considerations 


SE 

ORGANIZATIONAL 

INTEGRATION 

SYSTEM 

ENGINEERING 

PROCESS 

INTEGRATION 


Design  Considerations 

This  section  describes  any  design  considerations  that  must  be  integrated  into  the 
engineering  design  effort  including  any  special  constraints  that  must  be  considered. 

^lease  enter  any  design  constraints. 


These  design  constraints  are  any  special  considerations  that  must  be  taken  into 
account  before  they  are  integrated  into  the  project  during  the  engineering  process.  The 
tej^t  should  also  describe  the  basis  for  these  design  constraints  and  how  the  technical 
authority  is  going  to  be  engaged  in  considering  and  integrating  these  constraints. 

Some  ef^amples  of  design  constraints  are  as  follows; 

*  The  system  shall  be  able  to  operate  using  the  three  phase  power  available  on 
board  a  ship. 

♦  The  system  shall  be  able  to  fit  into  a  standard  19"  rack. 

While  these  constraints  look  like  requirements,  they  are  not  system  requirements 
because  they  do  not  specify  what  the  system  must  do,  nor  do  they  specify  how  well 
the  system  must  perform  a  capability;  they  constraint  the  possible  solutions  by  limiting 
the  choices  available  to  the  engineers,  and  are  therefore  design  requirements  that 
constrain  the  solution  space. 


The  nature  of  the  SEP  requires  more  open  input  text  fields,  but 
EPB  helps  by  providing  elaborations  and  examples  for  the  user 
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SEP 

NAS  Pensacola 
□  SP  Survey 

^  PROJECT  SETUP 

Ef  DOCUMENT 
™  SETUP 

^  PROGRAM 

^  INTRODUCTION 

n  ACQUISITION 
HISTORY 

Ef  SYSTEM 

CAPABILITIES 

sr 

ORGANIZATIONAL 

INTEGRATION 

SYSTEM 

ENGINEERING 

PROCESS 

Planning 

Process 

Improyement 

Modeling  and 
Simulation 

Resources 
Trade  Studies 


gf  INTEGRATION 
^  INTEGRATED 


0  SO  100 

Trade  Studies 

This  section  should  include  a  brief  description  of  the  process  used  to  determine  trade-offs 
between  various  attributes  of  the  program  (e.g.^  between  requirements  and  design), 
Information  about  how  trade  studies  are  addressed  within  the  organization  will  be 
automatically  embedded  into  the  document,  To  view  the  embedded  information  about  how 
trade  studies  will  be  addressed^  click  the  "Click  to  view  the  embedded  trade  studies  tef^t" 
link  below, 

10  Click  to  view  the  embedded  trade  studies  text. 

Trade  studies  will  be  addressed  in  accordance  with  the  SSC-C  Technical  Solutions 
Process  Manual  and  SSC-C  Decision  Analysis  and  Resolution  Process  Manual  where  the 
development  of  alternate  solutions^  selection  criteria  and  trade  processes  are  discussed. 


The  actual  trade  studies  to  be  performed  on  the  program  will  be  captured  and  listed  in  the 
control  below. 

Please  enter  the  trade  studies  that  will  be  conducted  on  this  program. 

Trade  Study 

Research  on  OSP  topologies 


Trade  Study 

Research  on  different  conduit  installation 
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PROJECT  PL  AMONG 


CM]Vn®-SE/SW 

Goal/Practice 

Number 


Compliance  matrix 
cross  references 
CMMI®  practices 
with  associated 
SSC-C  Process 
Manuai  and  Project- 
specific  plan 

(No  matrix  for  SEP) 


1 


pp  1.1 


PP  1.2 


PP  1.3 


PP  1.4 


PP  2 


CMMI®-SE/SW  Level  2 
Process  Aiea 
Project  Planning  (PF) 


Est ab li s h  E stim ate s .  Estim ate s  of  pr oj e  ct 
planning  parameters  are  established  and 
maintained. 


Estimate  the  Scope  of  the  Project.  Establish 
and  maintain  a  top-level  v^ork  breakdov^n 
structure  (WBS)  to  estimate  the  scope  of  the 
project. 


Establish  Estimates  of  Project  Attributes. 
Establish  and  document  estimates  of  the 
attributes  of  the  work  products  and  tasks. 


Define  Project  Life  Cycle.  Define  the  project 
life  cycle  phases  upon  which  to  scope  the 
planning  effort. 


Determine  estimates  of  Effort  and  Cost. 
Estimate  the  project  effort  and  cost  for  the 
attributes  of  the  work  products  and  tasks 
based  on  estimation  rationale. 


Develop  a  Project  Plan.  A  project  plan  is 
established  and  maintained  as  the  basis  for 
managing  the  project. 


SSC-C 
pp  Process 
Manual 
Paragraph 


3.2 


3.2 


3.2 


3.2 


3.2 


3.3 


593 

PMP 

Paragraph 


1.2.1 


1.2.1 

3 

Appendix  A 


1.2.1 

1.3 


1 

1.2.1 


1.3 

1.2.1 

Appendix  A 


1 
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ational  Measurement  Repository  (OMR) 


AOrganizational  database  for  collecting  standard 
project  measures  and  providing  analysis 

ACurrently,  the  OMR  accepts  the  following  standard 
project  measures 


Category 

Core  Measure 

Schedule  Performance 

A  Estimated  vs.  Actual  Milestone  dates 

A  Estimated  vs.  Actual  Monthly  Task 
completions 

Cost  Performance 

A  Estimated  vs.  Actual  Milestone  costs 

A  Estimated  vs.  Actual  Monthly  costs 

Process  Performance 

A  Total  #  of  noncompliance  issues 
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OMR 

Datastore 


OMR  Client 
Application 


"I 


Organizational  Performance 
&  Analysis 


Population  Size: 

Mean: 
Medi an : 


172 

-21.22% 
-6.22% 
Mode:  0.00% 
Min:  -163.12% 
Max:  330.77% 
Variance:  71.SS% 
Standard  Deviation:  84.765^5 
Probability  of  X  <  Min:  A. 75% 
Probability  of  X  >  Max:  0.00% 
Probability  of  Min  <  X  <  Max:  95.25% 
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AProvides  interface  for 
input  and  query  functions 

ACenerates  quarteriy 
organizationai  report 

AProjects  can  use  to 
manage  own  projects 

V  Capture  standardized  cost, 
schedule,  and  process 
performance 

AOMR  implementation 
included  hands-on 
training 

ALaying  the  groundwork  for 
higher  maturity 


OMR  Application 


iOl  OMR  (Metrics  Repository)  Application  VI  .0.6 
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Category 

Core  Measure 

Cost  Performance 
(More  granularity) 

A  Government  vs  Contractor  budget 

1  ODC 

1  Travel 

1  Training 

1  Materials 

Quality 

A  Peer  Reviews 

1  Effectiveness 

1  ROI  (hours  expended  vs  hours  saved) 

A  Pre-Deployment  Defect  Detection/Prevention 

1  Defect  decrease  for  successive  phases 

1  PITCO  vs  SOVT  defects 

A  Post-Deployment  Defects 

Need  improved  project  and  organizational  measures  to 
address  Maturity  Level  4/5  requirements 
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Appraisal  Wizard  Tool 
Used  for  SCAMPI  Appraisals 


%  Element  Review  (AM009)  Element:  PP  5P  2.1-1 


■i  Options  Record  View 


AOesigned  for 
CMMI 
appraisals 

ALink  to  project 
documents 

AEasy  to 
configure 

ACaptures  team 
comments 

Aimproves 
efficiency  of 
appraisal  team 


:  Model 


:  Document  Filtering Document  List - 

’  Rating  Level  Appraisal 
^State  I  H 


^  Element  Colo 


I  Element  Records  |  Element  Documents  Document  List 


'fl 


Appraisal 

*^*Wizard 

v7 


Drag  a  column  header  here  to  group  by  that  column 


Drag  a  column  header  here  to  groL 


ii; 

Element 

Parc 

PPSP  1.1-1 

PP 

PPSPT.?'! 

[P^ 

PPSPT.3-1 

PPSP  1.4-1 

PP 

PPSP  2.1-1 

pspa2-i 

pp 

[pF^a.si 
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I  Rec  ID  Record  Ty|  Status 


1  107 

Compliant  |/ 

667 

Compliant 

385 

Complia^^Jd 

1346 

Compliant 

4727 

►:  4330 

Co 

538G 

Co 

Verifica  Record  Text 

Yes  ICIS  POAMS  and  Budget  reports  are  direct  OE  tf* 


CICS  Schedule  FY2007  and  Budget  breakdown  for  OG  are  direct  OE.  Funding  summary  provides 


MS  Project  schedules  for  various  years  including  FY07  plan;  Requirements  Baselines  show  training 


Monthly  JTWS  Financial  Report  ■  March  2006  and  JTWS  Master  Schedule  are  direct  OE  that  budget 


□ 

□ 


Practice 
Characterizations 
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Record  Fields  Elements  /  Projects  /  T earn  Members  /  Data  Sources  |  Record  Documents  I 


Create  New  Document 


Individual  Project 
Records  per 
Practice 


S  P  2. 1  -1  E  stablish  the  B  udget  and 
Schedule 

Establish  and  maintain  the  project's 
budget  and  schedule. 

The  project's  budget  and  schedule  are 
based  on  the  developed  ^^^ates  and 
ensure  that  budget  allocatiX 
complexity,  and  task  depend 
appropriately  addressed. 


Drag  a  column  header  here  to  group  by  that  column 


Evidence  T  Doc  Type  Owner 


Comments  (Di 


Specific 

Practice 

Description 


Evidence  List  by 
Practice  &  Project 


Appraisal  Wizard  is  a  product  from  Integrated  Systems  Diagnostics,  Inc. 

http://www.isd-inc.com 
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Aj-day  Introduction  to  CM  Ml®  course 
teaches  the  full  CMMI®  model 

I  Students  learn  how  the  best  practices  build  and  delate 
across  process  areas 


T  Learn  the  terminology 

^El-Authorized  instructors  are  well-versed  in 
our  impiementation  to  augment  materiai  with 
SSC-C  specific  content 


I  Highlight  SSC-C  tools  and  resources 
I  Actively  involved  in  projects,  teams,  and  infrastructure 

ADver  350  employees  trained 


T  Want  to  build  a  cultural  foundation  within  the 
engineering  departments 
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3-day  on-site,  classroom  course  I 

T  Based  on  SMU  SE  Masters  course 
T  Customized  to  incorporate  SSC-C  SE  process  ; 
T  Over  340  SSC-C  engineers  trained 

1-day  SE  for  Managers  course  added 

T  Over  60  SSC-C  managers  trained 


“It  was  extremely  beneficial  to  have  a  professor  with  extensive 
knowledge  of  the  subject  matter  and  one  who  could  apply  it  to  the 
SPAWAR  methods/’ 

“The  most  positive  aspects  I  took  from  the  class  was  the  visual 
correlation  with  what  was  asked  for  and  what  was  produced.  ” 

“I  would  recommend  it  to  all  the  program  leads/engineers.” 
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New  On-Site  Courses 


ARisk  Management 

T  Piloted  in  September,  2007 
A4-day  course 

V  Designed  for  Risk  Managers  or  Project  Managers 

AEngineering  Project  &  Process  Mgmt  Workshop 
(aka  SE  Process  Improvement) 

T  Focus  on  how  to  use  the  SSC-C  processes  on  your  project 
AUsing  ePIan  Builder  to  develop  plans 
AHow  to  establish  your  CM  and  PPQA  procedures 

V  Round  2  of  curricuium  review  completed  in  September 

AOuality  Assurance  (FY2008) 

V  initiai  discussions  heid  with  ASQ  certified  instructor  to  tailor 
course  for  Quality  Managers  at  the  project  ievei 
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0  Based  Training  (WBT)  Moduies 


AOeveloped  to  directly  meet  SSC-C’s  needs 

T  Embedded  links  directly  to  SSC-C  documents  and  SOPs 

V  DAL)  too  ACAT -level/large  program  oriented 

AWBTs  feature  extensive  branching  and  rollovers 

V  Better  course  flow  and  maintains  interest 

V  Provides  more  detail  for  those  interested 


AAudio  summary  on  many  pages 
ABookmark  progress  -  come  back  later 
ACourses  developed  to  be  NMCI  and  508  compliant 

V  Utilize  HTML,  JavaScript,  and  ASP  pages  with  SQL  Server 
database 


T  Designed  for  Internet  Explorer  (5.5  +),  Flash  (5.0  +),  Windows 
Media  Player  (9.0  +) 
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Introduction  to  Systems  Engineering 


T  10-module  web-based  training  (~16  hours) 

V  Closely  aligned  to  SSC-C  SE  Process,  SE  Fundamentals  Course, 


ISO/IEC  15288  and  IEEE  standards 


ISO/IEC  15236 
Technical 
^  Processes 


SSC-C 

Standard  Processes 
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I  Includes  hotlinks  to  referenced  documentation 
A  Process  manuals,  policies,  standards 
AOreat  for  Topic-specific  refresher  training 
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Risk  Management  WBT 


Alopics 

V  Risk  identification 

V  Analysis  tools  and  techniques 

V  Mitigation  planning 

V  Risk  monitoring 

ASection  Test  Questions 

AHot  Links  to  Examples 

V  SSC-C  Formats 

T  Project  Risk  Reports 
I  Tools 

V  DAL)  /  External  resources 


Continuous  Risk  Management  Process 


Click  on  each  graphic  link  to  view  its 
description. 


More  relevant  and  understandable  for 
SSC-C  than  the  DAU  module 
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Architecture  Development  WBT 


Aintroduction  to  Architecture  Development  and  DoDAF 

T  Designed  to  educate  and  promote  value  of  system  architecture  to 
non-architects  and  new  engineers 

T  Tests  for  understanding  after  each  section 
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What  We  Have  Accomplished 


AProcess  Focus 

I  Defined  Policies  and  Processes 
I  Aligned  with  DoD  and  SPAWAR  guidance 
T  Aligned  with  industry  standards  and  CMMI®  model 
T  Built  organization  structured  around  processes  and  process  improvement 

At raining  is  Criticai 

T  Providing  Fundamentals  of  Engineering  for  new  and  old  professionals 
T  Developed  web-based  training  for  feelf-pacedoand  refresher  training 
T  Defining  a  structured  technical  career  development  path  for  engineers 

AToois  for  the  Engineers 

T  Developed  ePIan  Bu/Vder  application  to  generate  planning  documents 

T  Developed  templates,  checklists,  and  web-based  document  repositories 
to  link  standards  and  DoD  guidance  to  day-to-day  tasks  and  processes 


Early  and  persistent  Systems  and  Software  Engineering 
appiied  to  programs  and  projects 
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ASenior  Management  support  is  critical  to  success 
Alraining 

I  Everyone  needs  to  be  engaged  i  ftrain  the  masseso 
I  Specific  training  for  process  owners/subject  matter  experts 

Autilize  Teams  (IPTs)  as  champions  of  specific  processes 

T  Multi-department  representation 
T  Change  agent  mentality 
T  Process-focused  charters 

AResource  Properly 

T  Implement  with  projects  that  want  to  improve,  can  benefit  from  efforts, 
and  that  recognize  own  weaknesses 

T  EPO  staff  provided  skilled  coaching,  resources,  support,  and  tools 

T  Project  members  learned  by  doing  and  maintaining 

AOoals  and  Publicity 

T  Keep  goals  to  sizable  bites  (projects) 

T  Publicize  successes;  Share  best  practices 
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ARecognition  of  SE  and  CMMI  effort 

T  1®^  SPAWAR  Systems  Center  to  achieve  Maturity  Level  2  (2005) 
T  1®^  SPAWAR  Systems  Center  to  achieve  Maturity  Level  3  (2007) 
T  Multiple  presenter  at  NDIA  SE  and  CMMI  conferences 
AHigh  interest  in  Tools,  Training,  and  Implementation 
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ABusiness  Results 

T  SCN:  ffhey  see  us  as  a  model  and  want  to  increase  our  efforts.6 

T  Automation  Program:  fWe  had  hundreds  of  sites  and  there  was  a 
need  for  a  structured  organization  to  put  a  ovrapperoaround  that 
and  control  it.  CMMI  became  the  wrapper.6 

T  CICS:  fCMMI  was  key  to  achieving  the  project  goal. 6 

T  VIDS:  fThe  VIDS  failure  (2000)  motivated  implementing  CMMI 
because  the  team  needed  to  change  course  or  the  customer 
would  have  no  confidence  in  system  development.  It  was  a 
tremendous  successe  6 

Aothers  Asking  for  Help 

T  PMS  408  T  CREW  program 

T  SESG/NAVAIR/NAVSEA 

T  Marine  Corp  T  Quantico 

T  Air  Armament  Center,  Eglin  AFB 
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Aincrease  usage  of  tools  across  departments/projects 
Mdd  additional  plans  to  ePIan  Builder  as  needed 
AContinue  internal  CMMI  Level  3  mini  assessments 
AEnhance/Expand  OMR 

ACommand  and  Department  Project  Reviews  process 

T  Look  at  quality  of  plans  and  implementation  of  best  practices 

T  Reviews  of  project  status  by  management  driven  by  project 
metrics 

T  More  Peer  Reviews  to  measure  reaveso 

ABetter  taiioring  guidance  for  smalier  projects 


Begin  Maturity  Level  4/5  implementation 
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Any  Questions? 


Contact  Infonnation: 

Michael  T.  Kutch,  Jr. 

SPAWAR  Systems  Center  Charleston 
Email:  michael.kutch@navv.mil 
Phone:  843-218-5706 


Mike  Knox 
TECHSOFT,  Inc. 

Email:  miknox@techsoft.com 
Phone:  850-469-0086 
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•  To  introduce  Functional  Capabilities  (FCs)  as  a  “useful” 
mechanism  for  managing  work  in  a  complex  product 
development  environment 

-  An  efficient  way  to  communicate  functionality  to  the  user,  the 
developer,  and  other  stakeholders 

-  A  structure  of  discrete  artifacts  and  flows  that  define  product 
development  lifecycle  activities 

•  logical  design 

■  system  analysis,  design  and  implementation 

■  testing 

-  A  scheme  for  planning,  tasking,  and  tracking  work 

-  An  effective  generator  of  artifacts  for  CMMI 

•  To  share  experiences  gained  from  initial  deployment  of 
this  project  management  process 
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ability  -  Context 


Consider  your  Program  to  be  a  large 
amount  of  functionality,  expressed 
as  capabilities 

Funcf/ona/ decomposition  will 
provide  increments  of  work  to  be 
accomplished,  resulting  in 
incremental  capability 

We  are  proposing  functional 
capabilities  as  a  project 
management  scheme  to  help 
deliver: 

A  the  right  product 

A  delivered  on  time  and 
within  budget 


COMPONENT 

SPECIFICATIONS 

TEST  CASES 

FUNCTIONAL 

FUNCTIONAL 

DESIGN 

ARCHITECTURE 

SME  Wish  List 

Threads 

IDD 


ECPs 


TECHNICAL 

ITERATION 


COMMUNICATION  OF 
CAPABILITY 


FUNCTIONAL 

CAPABILITY 


INCREMENTAL 

TASKING 


PROJECT  CONTROL 
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•  Problem  Statement 

•  SIAP 

•  Program  Performance 

•  Functional  Capability  Overview 

•  Functional  Capability  Elaboration 

•  CMMI  Mapping 

•  Summary 
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•  Product  developers  routinely  fail  to  execute 
their  projects 

-  GAO  Report  05/301 , 2005 

-  Defense  Acquisition  Performance  Assessment,  2006 

•  How  do  acquirers  gain  insight  into  their  project’s 
performance? 

-  Does  developer  CMMI  ML  significantly  affect  project 
performance?  If  not,  why  not? 

•  How  do  contractors  know  they  are  producing  what 
their  customer  wants? 

•  Do  we  need  a  different  project  context  for  Systems  of 
Systems  (SoS)? 

-  CMU/SEI-2006-TR-017,  Systems  of  Systems:  “Scaling  Up  the 
Development  Process” 
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riunication  of  Capability 


•  Capability  must  be  expressed  in  user  terms... 

What  they  want 

-  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  is  not  sufficient 

-  systems  engineers  need  more  expressive  methods  for 
requirements  capture  and  development 

•  What  they  will  get 

-  “System”  specifications  (to  drive  developers)  that 
users  can  relate  directly  to  capabilities 

•  And  how  they  know  they  are  getting  it 

-  Earned  value  expressed  in  terms  of  capability,  i.e., 
“earned  capability” 

■  performance-based  earned  value 

■  assessment  of  functionality  bow  wave 
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•  SoS:  Collaborating  systems  developed  by 
collaborating  system  acquisition  teams 

V  highly  autonomous  systems  and  teams 

•  Process  challenges  in: 

-  organizational  ownership,  responsibilities,  and  technical 
team  interactions 

-  systems: 

■  boundary  definition 

■  iegacy  systems  and  continuous  technoiogy  evoiution 

■  continuous  capabiiity  evoiution 

-  project  definition,  measurement,  and  reporting  mechanisms 

-  project  execution  processes 

•  Practical  process  methods  are  needed 
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ingle  Integrated  Air  Picture 


A  FCs  developed  from  experiences  in  SIAP 


6  SIAP  is  a  Software  Intensive  System 
6  FCs  should  apply  to  SoS  in  general 


A  SIAP  Capability 


6  user  viewpoint:  common,  correct,  complete,  continuous,  timely  track 
situation  presentation 

— system  viewpoint:  state  of  data  consistency  among  distributed, 
replicated  data  stores,  for  objects  of  peer  interest 


DISCLAIMER:  This  presentation  makes  no  statement  concerning  current  SIAP  engineering  practices. 
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ity  Material  Challenge 


A  SIAP  requires  interactions  of  networked  peers,  each 
an  operational  node  hosting  multiple  integrated 
systems 

A  Network  connections  are  weak,  with  ad  hoc,  dynamic 
configurations 


Project  Management  by  Functional  Capability 

Fred  Schenker  and  Bob  Jacobs,  November  15,  2007  9 

©2007  Carnegie  Mellon  University 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


ity 


Material  Solution 


-  Executable  Object  Model  transformable  to  code,  with 
core  required  functionality 

-  Agile-development  processes 


BECOMES 


Unpredictable  Heterogeneous 
Set  of  Systems 


Predictable,  Logically 
Homogeneous  Federation 
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•  Functional  Capabilities  express  functional 
requirements 

-  manageable  abstraction  level  for  SoS 

-  meaningful  to  user  and  developer 

•  An  FC  identifies  a  value-chain 

-  tangible  artifacts 

-  framework  for  measuring  program 
process  performance 

•  An  FC  represents  value  that  can  be  earned 
against  a  planned-performance  baseline 

-  an  example  of  Performance-Based  Earned  Value® 
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Establish  relative  size  measures  for  each  capability 
Establish  dependencies  between  capability  projects 
Establish  the  approved  list  of  capability  (or  value) 

Release  work  as  appropriate  and  accrue  “value”  against  the  project 
capability  “baseline”  at  Management  reviews  Capabil- 

Measure  project  lifecycle  task  duration  and  effort  to  refine  estimation  o-meter 
process  and  establish  project  historical  parametric  data 
Capability  can  be  “re-scoped”,  but  deviations  from  the  baseline  are  easily 
recognizable  as  the  “bow-wave”  of  functionality 
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ability  Life  Cycle 


•  Each  FC  advances  through  lifecycle  phases, 
representing  states  of  completion,  defined  by  artifacts 

•  Artifacts  are  reviewed  at  Quality  gates,  providing 
evidence  of  value 
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•  Early  in  the  Program  Lifecycle,  Functional  Capability 
planning  definitions  are  needed: 

-  Based  on  End-to-End  mission  scenarios 

-  No  more  than  one  or  two  pages  per  FC 

-  Preliminary  allocation  of  requirements 

-  High-level  textual  description 

-  Basis  of  estimates  for  effort,  resource,  and  schedule 
planning  (use  cases,  complexity,  requirements,  etc.) 

-  Use  historical  data  where  possible  (and  practical) 

-  Establish  FC  priority  and  FC-FC  dependencies 

•  Use  the  planning  definitions  to  establish  Earned 
Capability  baseline  and  to  scope  project 
deliverables  and  dates 
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•  Refine  the  scenarios  to  specify  the  capabilities 

•  Finalize  allocation  of  functional  requirements  to  the 
notional  FC 

•  Elaborate  the  FC 

-  Create  a  contextual  description  of  the  functionality 

-  Create  sequence  diagrams,  use  cases,  behavior  diagrams 

-  Ensure  the  allocated  requirements  are  explained 
adequately  in  the  context  of  the  functionality 

-  Provide  criteria  for  FC  acceptance 

•  Validate  the  FC 

-  Peer  review 

-  Customer  review 

-  Management  review  (Q-Gate) 
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•  Start  with  validated  functional  design 

•  Allocate  functionality  to  legacy  components 

-  Identify  and  analyze  design  alternatives  as  necessary, 
especially  for  risk  mitigation 

-  Update  existing  /  create  new  design  documentation, 
component  specifications 

-  Create  work  packages  to  implement  the  new  designs 

-  Update  previous  estimates  of  effort  and  schedule 

-  Identify  task  dependencies,  establish  need  for  commitments 
for  inter-component  deliverables 

•  Validate  the  Analysis 

-  Peer  review 

-  Customer  review 

-  Management  review  (Q-Gate) 
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•  Start  with  Functional  Capability  Definition 
Document  requirements  acceptance  criteria 

-  Review  the  acceptance  criteria 

■  New  scenarios  that  need  to  be  instantiated 

■  New  requirements  that  need  to  be  verified 

■  Legacy  requirements  that  have  been  further  clarified 

-  Develop/modify  test  cases  based  on  the  criteria 

-  If  necessary,  create  new  scenario  (data  set) 

-  Identify  need  for  additional  test  tools,  and  develop 
those  tools 

•  Validate  the  Test  Preparation 

-  Peer  review  test  cases  and  scenarios 

-  Management  review  (Q-Gate) 
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•  Start  with  validated  System  Analysis 

•  Coordinate  the  tasks  so  that  the  Functional 
Capability  is  achieved 

-  Identify  and  negotiate  commitments  between 
development  teams 

-  Establish  development  goals  for  the  next  increment  of 
production  (TimeBox) 

-  Execute  tasks  in  accordance  with  the  plan 

-  Perform  verification  tasks  and  pass  on  to  integration 

•  Integrate  the  new  products 

-  Check  interfaces,  build  new  integrated  product 

-  Verify  new  build  (smoke  test) 

•  Validate  the  Development  and  Integration 

-  Management  Review  (Q-Gate) 
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•  Start  with  stable  production  build 

-  Regression  test  (with  new  test  cases) 

-  Log  bugs/defects 

-  Perform  SoS  simulated  testing  (if  possible) 

-  Evaluate  performance  bottlenecks;  potential  SoS 
issues 

-  Produce  test  report 

•  Validate  the  results 

-  Management  review  (Q-gate) 
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MMI 


Q:  So  what  does  this  have  to  with  CMMI  anyway? 

This  is  the  CMMI  User’s  Conference,  right? 

A1:  If  you  adopt  the  Functional  Capability  lifecycle, 
you  get  a  lot  of  CMMI  credit... 


A2:  If  you  managed  your  projects  this  way  you 
could  use  CMMI  practices  (esp.  M&A)  to  help 
you 

-  Produce  what  your  customers  want 

-  Make  sure  your  contractor  is  performing 
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•  Project  Planning  (SG  1 ,  SG  2,  SG  3) 

-  Estimation  of  FC  scope  (size,  complexity,  effort,  priority) 

-  Standard  FC  WBS 

-  Defined  FC  lifecycle 

-  FC  implementation  risks 

-  Stakeholder  identification  and  involvement 
(FC  prioritization) 

-  FC  Implementation  Budget  and  Schedule 
(FC  Owners  «  CAMs) 

-  Summation  of  FC  Planning  Definitions  (Baseline  Plan) 

-  Commitments  established  between  IPTs 

•  Project  Monitoring  and  Control  (SG  1) 

-  Defined  project  milestones  (Q-Gates) 

-  “Earned”  Capability  to  calibrate  program  performance 
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•  Requirements  Development  (SG  1,  SG  2,  SG  3) 

-  Stakeholder  “needs”  documented  (or  referenced)  in 
FCDD,  and  validated  via  peer  review 

-  Context  for  requirement  implementation  and  acceptance 
criteria  provided  in  FCDD 

■  Basis  for  product  component  and  interface  requirements 

■  Definition  of  required  functionality 

■  Basis  for  requirements  validation 

-  Use  cases  documented  in  the  FCDD  (Operational 
concepts  and  scenarios) 

•  Technical  Solution  (SG  1,  SG  2,  SG  3) 

-  Alternative  solutions  documented  in  FCDD  and 
propagated  through  System  Analysis  of  FC 

-  FCDD  represents  documentation  of  Functional  design 
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abilities  -  CMMI  Mapping 


•  Requirements  Management  (SG  1) 

-  FCDD  helps  to  develop  an  understanding  of  requirements 

-  FCDD  to  Requirements  trace  useful  for  identifying  impact 
of  changes 

•  Verification  (SG  1 ,  SG  2,  SG  3) 

-  Requirements  Verification  acceptance  criteria  defined  in 
FCDD 

-  Defined  artifacts  represent  obvious  opportunities  for  Peer 
Review 

•  Validation  (SG  1 ,  SG  2) 

-  Defined  artifacts  are  used  to  interpret,  communicate  and 
validate  product  design 

-  Product  lifecycle  defines  artifacts,  essential  for  planning 
validation  activities 
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•  Integrated  Project  Management  (SG  2) 

-  FC  Definition  Document  provides  basis  for  management 
of  stakeholder  involvement,  dependencies,  and 
identification  (and  resolution)  of  coordination  issues 

•  Measurement  and  Analysis  (SG  1,  SG  2) 

-  FC  baseline  represents  program  commitment 

-  Tracking  of  FC  progress  connects  tasks  execution  to 
management  information  needs 

•  Quantitative  Project  Management  (SG  1 ,  SG  2) 

-  FC  baseline  represents  the  program’s  performance 
objective 

-  Tracking  of  FC  progress  helps  to  determine  whether  the 
program’s  objectives  for  performance  are  being  satisfied, 
and  are  used  to  identify  appropriate  corrective  actions 
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•  Functional  Capability  provides  a  useful  framework 
for  managing  projects 

-  In  a  complex  environment  (SoS) 

-  As  a  significant  contributor  of  value-adding  artifacts 

-  As  a  starting  point  for  introducing  quantitative  methods 
into  the  project  management  process 

-  As  a  means  of  communicating  capability,  both  desired 
and  earned 

-  As  an  effective  means  to  deliver  relevant  technical  and 
project  management  content  to  external  stakeholders 

-  As  a  method  of  assessing  the  “bow-wave”  on  a  project, 
and  calibrating  the  reported  earned  value 
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■  A  demonstrated  capability  of  an  aircraft  to  function 
satisfactorily  within  established  limits 

■  Military  Certification  vs.  Civil  Certification 

□  Militarily  qualification  requires  demonstration  of 
airworthiness  to  protect  crew  and  passengers 

□  Civilian  certification  concentrates  on  safety  of  everyone 
else 
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years 

□  Spain  grounded  aircraft  until 
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□  Singapore  request  data  6  years 
after  delivery 
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Your  CMMI  and  improvement  expertise  is  very  relevant! 
You  can  help! 
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and  Technology) 

T  CSAB  (nee  the  Computing  Sciences  Accreditation 
Board) 

A  The  ABET  accreditation  process 
A  Accreditation  criteria 

A  Status  of  accreditation  of  disciplines  of  interest 
A  Government  and  industry  practitioners 

T  ABET  and  CSAB  want  you! 
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I  relevance 
T  accountability 

A  Answers  to  important  questions 


T  How  can  employers  judge  preparation  of  graduates? 

I  How  can  students  choose  appropriate  programs  and 
institutions? 

T  How  can  professions  guide  the  establishment  of  new 
programs  and  improve  current  programs? 
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in  Educational  Approach 


A  Traditional  approach  to  science  and  engineering 
education 

V  Emphasis  on  curricula 

A  how  students  are  educated 

V  Culture  of  independence  among  faculty 

A  Target  approach  for  science  and  engineering 
education 


T 

T 

T 


Emphasis  on  outcomes 

A  what  knowledge,  skills,  abilities  graduates  possess 

Emphasis  on  continuous  improvement  based  on 
measurement  and  assessment 

All  this  requires  greater  coordination  among  faculty 


ABET  is  a  key  actor  in  furthering  this  approach 
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A  established  in  1932 

T  incorporated  computing  accreditation  responsibility 
beginning  in  2001  (from  CSAB,  formed  in  1982) 

A  provides  a  mechanism  for  professional  societies 
to  examine  and  affect  academic  quality 

A  a  federation  of  31  technical  and  professional 
societies  representing  over  1 .8  million  technical 
professionals 

A  accredits  applied  science,  computing, 
engineering,  and  technology  programs 
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vviTy-iS  ABET  Accreditation 

Important? 


Parents  and  Students  . . . 

A  Look  to  accreditation  to  choose  the  right  study  programs. 
Employers  . . . 

A  Rely  on  accreditation  to  ensure  that  employees  are  qualified  to 
practice. 

Licensing  and  Certification  Boards  . . . 

A  Count  on  accreditation  to  screen  applicants. 

Colleges  and  Universities  . . . 

A  Use  accreditation  as  a  structured  mechanism  to  assess, 
evaluate,  and  improve  the  quality  of  their  programs. 

Graduate  Schools  . . . 

A  Check  accreditation  to  determine  the  eligibility  of  applicants. 
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A  CSAB  is  a  federation  of  the  ACM,  lEEE-Computer 
Society  and  Association  for  Information  Systems  for 
accreditation  issues. 

A  Formed  in  1982  for  accrediting  computing  programs 

A  Transferred  accreditation  mechanics  responsibilities 
to  ABET  beginning  in  2001 

A  Continues  on  as  the  “society”  representing  the 
member  societies  on  matters  of  accreditation. 

A  computer  science 
A  information  systems 
A  information  technology 
A  software  engineering 
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Board) 

)  A  The  ABET  accreditation  process 
A  Accreditation  criteria 

A  Status  of  accreditation  of  disciplines  of  interest 
A  Government  and  industry  practitioners 
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Composition 

A  Team  Chair 

A  Program  Evaluators  (PEVs)  (2  or  more) 

Team  Chair 

A  a  member  of  the  Commission 
A  appointed  by  the  Commission  Executive  Committee 
A  leads  the  Visit  Team 
A  interfaces  with  the  institution 
A  presents  the  findings  at  the  July  commission  meeting 

Program  Evaluators 

A  selected  by  their  member  societies  (CSAB  for  computing) 
A  provide  expert  knowledge 

A  evaluate  programs  according  to  evaluative  criteria 
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Obvious? 


Making  observations 


Comparing  observed  practices  against  standards 


Applying  professional  judgment 
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CTiteria  Categories 


1.  Students 

2.  Program  Educational  Objectives 

3.  Program  Outcomes 

4.  Continuous  Improvement 

5.  Curriculum 

6.  Faculty 

7.  Facilities 

8.  Support 

9.  Program  Criteria 
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3:  Program  Outcomes^ 


A  The  program  has  documented,  measurable  outcomes 
that  are  based  on  the  needs  of  the  program® 
constituencies. 

A  The  program  enables  students  to  achieve,  by  the  time  of 
graduation: 

(a)  An  ability  to  appiy  knowiedge  of  computing  and  mathematics 
appropriate  to  the  discipiine 

(b)  An  abiiity  to  anaiyze  a  probiem,  and  identify  and  define  the 
computing  requirements  appropriate  to  its  solution 

(c)  An  abiiity  to  design,  impiement,  and  evaluate  a  computer- 
based  system,  process,  component,  or  program  to  meet  desired 
needs 

(d)  An  abiiity  to  function  effectiveiy  on  teams  to  accompiish  a 
common  goai 
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3:  Program  Outcomes^ 


(e)  An  understanding  of  professional,  ethical,  legal, 
security  and  social  issues  and  responsibilities 


(f)  An  ability  to  communicate  effectively  with  a  range  of 
audiences 


(g)  An  ability  to  analyze  the  local  and  global  impact  of 
computing  on  individuals,  organizations,  and  society 

(h)  Recognition  of  the  need  for  and  an  ability  to  engage 
in  continuing  professional  development 

(i)  An  ability  to  use  current  techniques,  skills,  and  tools 
necessary  for  computing  practice. 
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4:  Continuous  Improvement 


A  The  program  uses  a  documented  process 
incorporating  relevant  data  to  regularly  assess 
its  program  educational  objectives  and  program 
outcomes,  and  to  evaluate  the  extent  to  which 
they  are  being  met. 


A  The  results  of  the  evaluations  are  documented 
and  used  to  effect  continuous  improvement  of 
the  program  through  a  documented  plan. 
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s  of  Specific  Interest  for 
This  Conference 


Computing  Accreditation  Commission  (currently 
three  program-specific  criteria) 

I  computer  science  (250  programs) 

I  information  systems  (30  programs) 

I  information  technology  (7  programs) 


Engineering  Accreditation  Commission  (currently 
nineteen  program-specific  criteria) 

I  software  engineering  (15  programs) 

V  system  engineering  currently  under  consideration 
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CDysierhrfngineering  Accreditation^ 


AiNCOSE  is  pursuing  admission  as  a 
member  of  ABET  with  the  intent  to  be  the 
lead  society  for  systems  engineering. 

A  The  ABET  Board  of  Directors  considered 
starting  the  ratification  process  during  its 
November  3,  2007  meeting. 

A  Accreditation  would  fall  under  the 
Engineering  Accreditation  Commission. 
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CDysierhrfngineering  Accreditation^ 


If  INCOSE  is  admitted,  it  will  need  to  address 
Program  Evaluator  responsibilities. 


Through  the  PAVE  (Partnership  to  Advance 
Volunteer  Excellence)  Project  common  support 
mechanisms  for  program  evaluators  exist  for 

I  a  program  evaluator  competency  model 
V  recruitment  and  selection 
I  training  and  evaluation 
T  reference:  http://www.abet.org/pave.shtml 
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Are  ABET  Program 
Evaluators? 


A  Deans 

A  Department  heads 
A  Faculty 
A  Industry  leaders 
A  Government  representatives 
A  Private  practitioners 


ABET  PROGRAM  EVALUATORS:  THE  FACE  OF  QUALITY  IN  TECHNICAL  EDUCATION 
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Muunniinal  Industry  Program 
Evaluators  Needed 


A  Practitioner  participation  is  criticai 

V  Where  did  the  emphasis  on  continuous 
improvement  and  outcomes-orientation  come 
from?  V  industry  inputs! 


A  The  Computing  Accreditation  Commission  is 
under-represented  in  industrial  participants 

V  1 0  industry/government  reps  out  of  47 
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Program  Evaluators  Do? 


A  Step  1 :  Review  the  self-study  report 

A  Step  2:  Visit  the  campus 

A  Step  3:  Decide  whether  the  program  meets 
the  criteria 

A  Step  4:  Travel  home  and  tie  up  loose  ends 


ABET  pays  travel  expenses 
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I  V  1  I  L 


Minimum  Qualifications 


for  Program  Evaluators 

1 .  Demonstrated  interest  in  improving  education 


2.  Membership  in  one  or  more  ABET  member  societies  or 
wiiiingness  to  become  a  member  prior  to  appiying  to  serve  as 
an  evaiuator 


3.  Formai  education  and  recognized  distinction  in  their  fieid 

a.  Program  evaiuators  with  an  industry  background  must 
possess  the  foiiowing: 

i.  Degree  appropriate  to  the  fieid 

ii.  Experience  in  empioyment  of  graduates  from  accredited 
programs 


ABET  PROGRAM  EVALUATORS:  THE  FACE  OF  QUALITY  IN  TECHNICAL  EDUCATION 
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teristics  of  Successful 
Program  Evaluators 

A  Technically  current 
A  Effective  at  communicating 
A  Interpersonally  skilled 
A  Team-oriented 
A  Professional 
A  Organized 
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7^  You  Qualified? 


A  Is  there  any  doubt  that  CMMI  and 
improvement  experience  is  an  excellent 
background? 


Raytheon 

itute  Integrated  Defense  Systems 

How  to  Apply 

1.  Begin  the  application  process  to  be  a  CS,  IS,  IT  or  SW  Engr  PEV  at 
http://www.csab.org/pev.htm* 

2.  If  accepted,  you  will  be  asked  to  complete  some  online  work  to 
prepare  for  formal  program  evaluator  training. 

3.  If  the  online  work  is  completed  satisfactorily,  you  will  attend  formal 
program  evaluator  training. 

4.  If  the  training  is  completed  satisfactorily,  you  will  be  approved  as  a 
program  evaluator.  In  some  cases,  you  will  be  asked  to  observe  a 
campus  visit  prior  to  approval  as  an  evaluator. 

5.  Based  on  your  availability  and  the  demand  for  program  evaluators 
in  your  field,  you  will  be  assigned  to  evaluate  a  program. 
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A  Additional  details  are  in  handouts 
A  Contact  information 

I  Larry  Jones:  lgj(g)sei.cmu.edu 
T  Dan  Nash:  j_Dan_Nash(g)raytheon. com 
I  Pat  LaMalva:  lamalva@csab.org 

A  Apply  at 

V  http://www.csab.org/pev.htm 
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Program 

Educationai 

Objectives 

Broad  statements  that  describe  the  career  and 
professionai  accompiishments  that  the  program  is 
preparing  graduates  to  achieve. 

(What  can  graduates  do  in  about  5  years  and 
continue  to  do  as  they  grow  professionaiiy?) 

Program  Narrower  statements  that  describe  what  students 

Outcomes  are  expected  to  know  and  be  able  to  do  by  the  time 

of  graduation.  These  relate  to  the  skills,  knowledge, 
and  behaviors  that  students  acquire  in  their 
matricuiation  through  the  program 
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CTiieria  Organization 


A  Students 

A  Program  Educational  Objectives 
A  Program  Outcomes 
A  Continuous  Improvement 
A  Curriculum 
A  Faculty 
A  Facilities 
A  Support 
A  Program  Criteria 
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and  Expanded  Features 


(^erion  1 :  Students 


Students  can  complete  the  program  in  a 
reasonable  amount  of  time.  They  have  ample 
opportunity  to  interact  with  their  instructors. 
Students  are  offered  timely  advising,  by  qualified 
individuals,  about  the  program’s  requirements 
and  their  career  alternatives.  Students  who 
graduate  from  the  program  meet  all  program 
requirements. 
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2:  Program  Educational 
Objectives 


The  program  has  documented, 
measurable  educational  objectives  that 
are  based  on  the  needs  of  the  program’s 

constituencies. 
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uriieriDri  3:  Program  Outcomes 


The  program  has  documented,  measurable 
outcomes  that  are  based  on  the  needs  of 
the  program’s  constituencies. 


The  program  enables  students  to  achieve, 
by  the  time  of  graduation: 


and  Expanded  Features  __ 

fc r I le r I orr  3 :  Program  Outcomes 


A 

A 

A 

A 

A 


(a)  An  ability  to  apply  knowledge  of  computing  and 
mathematics  appropriate  to  the  discipline 

(b)  An  ability  to  analyze  a  problem,  and  identify  and 
define  the  computing  requirements  appropriate  to  its 
solution 

(c)  An  ability  to  design,  implement,  and  evaluate  a 
computer-based  system,  process,  component,  or 
program  to  meet  desired  needs 

(d)  An  ability  to  function  effectively  on  teams  to 
accomplish  a  common  goal 

(e)  An  understanding  of  professional,  ethical,  legal, 
security  and  social  issues  and  responsibilities 


A 

A 

A 

A 
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3:  Program  Outcomes 


A  (f)  An  ability  to  communicate  effectively  with  a 
range  of  audiences 

A  (g)  An  ability  to  analyze  the  local  and  global 
impact  of  computing  on  individuals, 
organizations,  and  society 

A  (h)  Recognition  of  the  need  for  and  an  ability  to 
engage  in  continuing  professional  development 

A  (i)  An  ability  to  use  current  techniques,  skills, 
and  tools  necessary  for  computing  practice. 
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rion  4:  Continuous 
Improvement 

The  program  uses  a  documented  process  incorporating 
reievant  data  to  regularly  assess  its  program  educational 
objectives  and  program  outcomes,  and  to  evaluate  the 
extent  to  which  they  are  being  met.  The  results  of  the 
evaluations  are  documented  and  used  to  effect 
continuous  improvement  of  the  program  through  a 
documented  plan. 
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i  f  crCffCufC70  g  I  I 

dmerion  5:  Curriculum 


The  program’s  requirements  are  consistent  with  its  educational 
objectives  and  are  designed  in  such  a  way  that  each  of  the  program 
outcomes  can  be  achieved.  The  curriculum  combines  technical  and 
professional  requirements  with  general  education  requirements  and 
electives  to  prepare  students  for  a  professional  career  and  further 
study  in  the  computing  discipline  associated  with  the  program,  and 
for  functioning  in  modern  society.  The  technical  and  professional 
requirements  include  at  least  one  year  of  up-to-date  coverage  of 
fundamental  and  advanced  topics  in  the  computing  discipline 
associated  with  the  program.  In  addition,  the  program  includes 
mathematics  appropriate  to  the  discipline  beyond  the  precalculus 
level.  For  each  course  in  the  major  required  of  all  students,  its 
content,  expected  performance  criteria,  and  place  in  the  overall 
program  of  study  are  published. 
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CTlterion  6:  Faculty 


A  A.  Faculty  Qualifications 

Faculty  members  teaching  in  the  program  are  current  and 
active  in  the  associated  computing  discipline.  They  each 
have  the  educational  backgrounds  or  expertise 
consistent  with  their  expected  contributions  to  the 
program.  Each  has  a  level  of  competence  that  normally 
would  be  obtained  through  graduate  work  in  the 
discipline,  relevant  experience,  or  relevant  scholarship. 
Collectively,  they  have  the  technical  breadth  and  depth 
necessary  to  support  the  program. 
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CTlterion  6:  Faculty 


A  B.  Faculty  Size  and  Workload 

There  are  enough  full-time  faculty  members  to  provide 
continuity,  oversight,  and  stability,  to  cover  the 
curriculum  reasonably,  and  to  allow  an  appropriate  mix 
of  teaching,  professional  development,  scholarly 
activities,  and  service  for  each  faculty  member.  The 
faculty  assigned  to  the  program  has  appropriate 
authority  for  the  creation,  delivery,  evaluation,  and 
modification  of  the  program,  and  the  responsibility  for  the 
consistency  and  quality  of  its  courses. 
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CTiierion  7:  Facilities 


Institutional  facilities  including  the  library,  other  electronic 
information  retrieval  systems,  computer  networks, 
classrooms,  and  offices  are  adequate  to  support  the 
educational  objectives  and  outcomes  of  the  program. 
Computing  resources  are  available,  accessible, 
systematically  maintained  and  upgraded,  and  otherwise 
adequately  supported  to  enable  students  to  achieve  the 
program’s  outcomes  and  to  support  faculty  teaching 
needs  and  scholarly  activities.  Students  and  faculty 
members  receive  appropriate  guidance  regarding  the 
computing  resources  and  laboratories  available  to  the 
program. 
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criterion  8:  Support 


The  institution’s  support  for  the  program  and  the  financial 
resources  available  to  the  program  are  sufficient  to 
attract  and  retain  qualified  faculty  members,  administer 
the  program  effectively,  acquire  and  maintain  computing 
resources  and  laboratories,  and  otherwise  provide  an 
environment  in  which  the  program  can  achieve  its 
educational  objectives  and  outcomes.  Support  and 
resources  are  sufficient  to  provide  assurance  that  the 
program  will  retain  its  strength  throughout  the  period  of 
accreditation. 
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CTirenon  9:  Program  Criteria 


Each  program  must  satisfy  applicable  Program 
Criteria  (if  any).  Program  Criteria  provide  the 
specificity  needed  for  interpretation  of  the 
General  Criteria  as  applicable  to  a  given 
discipline.  If  a  program,  by  virtue  of  its  title, 
becomes  subject  to  two  or  more  sets  of  Program 
Criteria,  then  that  program  must  satisfy  each  set 
of  Program  Criteria;  however,  overlapping 
requirements  need  to  be  satisfied  only  once. 
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computer  Science 


A  3.  Program  Outcomes 

The  program  enables  students  to  achieve,  by  the  time  of 
graduation: 

(j)  An  ability  to  apply  mathematical  foundations, 
algorithmic  principles,  and  computer  science  theory  in 
the  modeling  and  design  of  computer-based  systems  in 
a  way  that  demonstrates  comprehension  of  the  tradeoffs 
involved  in  design  choices.  [CS] 

(k)  An  ability  to  apply  design  and  development  principles 
in  the  construction  of  software  systems  of  varying 
complexity.  [CS] 
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A  5.  Curriculum 

Students  have  the  following  amounts  of  course  work  or  equivalent 
educational  experience: 

a.  Computer  science:  One  and  one-third  years  that  Includes: 

1)  coverage  of  the  fundamentals  of  algorithms,  data  structures,  software 
design,  concepts  of  programming  languages  and  computer  organization 
and  architecture.  [CS] 

2)  an  exposure  to  a  variety  of  programming  languages  and  systems.  [CS] 

3)  proficiency  in  at  least  one  higher-level  language.  [CS] 

4)  advanced  course  work  that  builds  on  the  fundamental  course  work  to 
provide  depth.  [CS] 
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A  b.  One  year  of  science  and  mathematics: 

1)  Mathematics:  At  least  one  half  year  that  must  include 
discrete  mathematics.  The  additional  mathematics  might 
consist  of  courses  in  areas  such  as  calculus,  linear 
algebra,  numerical  methods,  probability,  statistics, 
number  theory,  geometry,  or  symbolic  logic.  [CS] 

2)  Science:  A  science  component  that  develops  an 
understanding  of  the  scientific  method  and  provides 
students  with  an  opportunity  to  experience  this  mode  of 
inquiry  in  courses  for  science  or  engineering  majors  that 
provide  some  exposure  to  laboratory  work.  [CS] 
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computer  Science 


A  6.  Faculty 
A.  Qualifications 

Some  full  time  faculty  members  have  a  Ph.D.  in  computer 
science. 
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mrormation  Systems 


A  3.  Program  Outcomes 

The  program  enables  students  to  achieve,  by  the  time  of  graduation: 

(j)  An  understanding  of  processes  that  support  the  delivery  and 
management  of  information  systems  within  a  specific  application 
environment  [IS] 
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A  5.  Curriculum 

Students  have  course  work  or  an  equivalent  educational  experience 
that  includes: 

a.  Information  Systems:  One  year  that  includes: 

1)  coverage  of  the  fundamentals  of  a  modern  programming 
language,  data  management,  networking  and  data  communications, 
systems  analysis  and  design  and  the  role  of  Information  Systems  in 
organizations.  [IS] 

2)  advanced  coursework  that  builds  on  the  fundamental 
coursework  to  provide  depth.  [IS] 

b.  Information  Systems  Environment:  One-half  year  of  coursework 
that  includes  varied  topics  that  provide  background  in  an 
environment  in  which  the  information  systems  will  be  applied 
professionally.  [IS] 

c.  Quantitative  analysis  or  methods  including  statistics.  [IS] 
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A  6.  Faculty 

Some  full-time  faculty,  including  those  responsible  for  the 
IS  curriculum  development,  hold  a  terminal  degree  in 
information  systems. 
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A  3.  Program  Outcomes 

The  program  enables  students  to  achieve,  by  the  time  of  graduation: 

(j)  An  ability  to  use  and  apply  current  technical  concepts  and 
practices  in  the  core  information  technologies.  [IT] 

(k)  An  ability  to  identify  and  analyze  user  needs  and  take  them  into 
account  in  the  selection,  creation,  evaluation  and  administration  of 
computer-based  systems.  [IT] 

(l)  An  ability  to  effectively  integrate  IT-based  solutions  into  the  user 
environment.  [IT] 

(m)  An  understanding  of  best  practices  and  standards  and  their 
application.  [IT] 

(n)  An  ability  to  assist  in  the  creation  of  an  effective  pro]ect  plan.  [IT] 
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rTTTunriation  Technology 


A  5.  Curriculum 

Students  have  course  work  or  an  equivalent  educational 
experience  that  includes: 

a.  Coverage  of  the  fundamentals  of 

1)  the  core  information  technologies  of  human 
computer  interaction,  information  management, 
programming,  networking,  web  systems  and 
technologies.  [IT] 

2)  information  assurance  and  security.  [IT] 

3)  system  administration  and  maintenance.  [IT] 

4)  system  integration  and  architecture.  [IT] 

b.  Advanced  course  work  that  builds  on  the  fundamental 
course  work  to  provide  depth.  [IT] 
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and  Software  Technology  Bode  Well 
for  the  Rapid  Adoption  of  CMMI 


CMMI  Technology  Conference  and  User  Group 
November  12-15, 2007 

Investigation,  Measures  and  Lessons  Learned  about  the 
Relationship  between  CMMI  Process  Capability  and  Project  or 
Program  Performance 
Hyatt  Regency  Tech  Center-  Denver,  CO 

Systems  and  Software  Technology  -  Enabling  the  Global  Mission 


Dr.  Kenneth  E.  Nidiffer 
Director  of  Strategic  Pians  for 
Government  Programs 
nidiffer@sei.cmu.edu 
703.908.1117 
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Federally  Funded  Research  and  Development  Center 
Created  in  1984 

Sponsored  by  the  U.S.  Department  of  Defense 

Locations  in  Pittsburgh,  PA;  Washington,  DC; 
Frankfurt,  Germany 

Operated  by  Carnegie  Meiion  University 
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Traditional  Future 


Standalone  systems 

Everything  connected-maybe 

Mostly  source  code 

Mostly  COTS  components 

Requirements-d  riven 

Requirements  are  emergent 

Focus  on  software 

Focus  on  systems  and  software 

Premium  on  cost 

Premium  on  value,  speed,  quality 

Stable  requirements 

Rapid  Change 

Control  over  evolution 

No  control  over  COTS  evolution 

Staffing  workable 

Scarcity  of  critical  talent 

trends  provided  by  Don  Reifer, 
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llenges:  Relationship  Between  Complexity 
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Software  is  Growing  in  Compiexity 

A80%  of  some  weapon  system 
functionality  is  dependent  upon  software^ 

AConsequences  of  software  failure  can  be 
catastrophic 

Software  Acquisition  is  Difficuit 

A46%  are  over-budget  (by 
an  average  of  47%)  or  late 
(by  an  average  of  72%)^ 

AfBuccessful  projectsohave 
68%  of  specified  features^ 

Software  is  Pervasive 

AIT  Systems,  C4ISR,  Weapons,  etc 


Standish  Group  CHAOS  Report 


2006 

2004 

2002 

2000 

1998 

1996 

1994 


0%  20%  40%  60%  80%  100% 

□  On-time  On  budget  □  Cancelled  □  Late  and  Over  buc^c 
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Platform 


Customer  Emphasis 


^  Enterprise 


Requirements 

Dominant 

Prime 


Acquisition  Modei 


Objectives 


Program  Execution 


.  strategic 
Teaming 


“Boxes”  - ►  “Layers  & 

Integration  Challenge  St3Cks” 

Proprietary  Plug  &  Play 

Architectures  and  Standards 

- 1 

The  emerging  dynamic  is  to  address  both  sides,  and  do  so  with  compressed 

delivery  schedules  via  improvements  in  systems/software  engineering 
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AA  shrinking  pool  of  experienced  workers. 

A  42%  decline  from  1990  peak  (AIA  Employment  Database) 

AConsolidation  left  our  industry  with  a  mature  workforce. 

A  54%  over  age  45  (BAH  Study) 

AEngineering  enrollment  trends  are  down. 

A  15%  decline  since  1991  (National  Science  Foundation 
Indicators) 

ABrutal  competition  for  technologists. 

A  Demand  for  experienced  engineers  is  projected  to  increase  by 
97%  between  1998  and  2008.  (US  Bureau  of  Labor  Statistics) 


A  key  challenge  is  how  to  transform  the  workforce  to  meet  demand 
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Pre  Boom 


Baby  Boom  Generation  X  Generation  Y 


r 


1940  1950  1960  1970  1980  1990 


Generation  Y  Characteristics 

What  Makes  Generation  Y  Tick 

^orn  late  1 970s  to  mid- 

AHigh  Expectation  of  Employers 

1990s 

AGoals,  Goals,  Goals 

A.arger  than  Generation  X 

AOesire  for  Immediate  Responsibility 

A/lore  ethnically  diverse 

ASalance  and  Flexibility 

Arechnoloqically  savvy 

Source:  Cara  Spiro,  DAL),  2006 
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Systems  Engr. 


System 

Design 


Systems 
Engineering  (SE) 


SW  Systems  Engr. 


Software  (SW) 
Requirements 
Analysis 


Systems  Engr. 


SW  Systems 
Engineering 


Architectural 

SW  Integration 

SW  Design 

Testing 

SW  Systems  Engr. 


SW  Engineering 


Detailed  SW 

SW  Subsystem 

Design 

Testing 

SW  Engineering 


Code  and 
Unit  Test 


I - h 

OSD  Initiative:  Integrated  Software  and  Systems  Engineering  Curriculum 

|| 
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^ect  (iSSEc) 


-Creating  a  Reference  Curriculum  for  Graduate  Software  Engineering  Education 

•iSSEc  is  sponsored  by  DOD  and  led  by  Stevens,  involving  4  sets  of  stakeholders: 

■  The  industrial  and  governnnent  workforce  who  are  the  customers  of  SWE  graduate 
education 

■  Academics  who  provide  SWE  and  SE  graduate  education 

■  Professional  societies  with  a  vested  interest  in  SWE  and  SE  graduate  education 

■  Government  organizations  who  fund  improvements  in  SWE  graduate  education 

-iSSEc  recognizes  that  the  divide  between  systems  and  software  engineers  in 
industry,  government,  and  academia  works  against  successfully  delivering 
modern  systems  in  which  software  is  almost  always  central. 

-iSSEc  will  integrate  SE  principles  and  practices  into  the  SWE  curriculum. 
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Demands/ 

Purposes 


Anticipated  Unanticipated 


Multiple 

Directed 

Distributed 

Collaboration 

Collaboration 

Autonomous 

Governance 

(Type  II  Agility) 

(Type  III  Agility) 

Entities 

Single 

Directed 

Directed 

V  n  It  n  Imlmiirf  ^ 

(Type  1  Agility) 

(Type  1  Agility  + 

Contingency 

Planning) 

Forms  of  Collaboration  from  “Architecting  Principles  for  Systems  of  Systems”,  by  Mark  W.  Maier 
http:/ /WWW.  in  feed,  com/open/papers/systems,  htm 
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2005  study  confirmed* *: 

Ain  advanced  knowledge-based  organizations,  management®  desire  for 
the  flow  of  knowledge  is  greater  than  the  desire  to  control  boundaries 

AUnlike  the  matrix  organization,  there  is  less  impact  on  the  dynamics  of 
formal  power  and  control 

Aimportant  to  measure  the  system  in  terms  of  user  performance 

*  Using  Communities  of  Practice  to  Drive  Organizational  Performance  and  Innovation,  2005,  APQ  study 


f^cquisitiono 


Advanced  Knowledge-Based  Organizations  (Big  A) 


▼ 


nacquisitiono 


From  fBcience  and  Technology  to  Support  FORCEn 

by  permission. 


Ref:  Jim  Smith,  (703)  908-8221 , jds@sei.cmu.edu 
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t:  Increased  Focus  on  Doing  More 


Random  motion  i  lots  of  energy,  Directed  motion  i  every  step  brings 
not  much  progress  you  closer  to  the  goal 


No  teamwork  T  individual  effort  Coordinated  efforts 


Frequent  conflict  Cooperation 

You  never  know  where  youdi  Predictable  results 

end  up 

I - 

Processes  Can  Make  the  Difference 
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VIM  and  CMMI  Technology  Transfer  Trends 


Intro  to  the  CMM  and  CMMI  Attendees 

(Cumulative) 


80000 1 


1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  YTD 

2007 


□  CMM  Intro  (discon’td. 
12/31/05) 

□  CMMI  Intro 

■  CMMI  Intermediate 


8-31-07 
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A  key  challenge  is  how  to  obtain  a  better  aiignment  of  risk  among  the 
key  stakehoiders  who  often  ieverage  technoiogy 
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Acceleration  of  Innovation  in  the  21st  Century 
ness  and  Society 


Ref:  Ray  Kurtzweil 
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Project  Management  (Especially  the  Acquirer)  to 
ively  Navigating  the  Green/Acquisition  Space 


Navigating  the  “Green  Space” 

FUsK-Ftewara  Hrererences 


High 


<U 


Low 


Increasing  gap 
between  Industry’s 
acceptable  risk/reward 
ratios  (dashed  line)  and 
the  reality  of  the 
marketplace  (solid  line) 


High 


The  “Green  Space” 
defines  the  area  where 
industry  initiatives  must 
provide  a  payoff  by 
reducing  risk  and/or 
increasing  reward. 


Low 


Risk 


Acquisition  changes  based  on  previous  legislation 
have  introduced  new  levels  of  risk. 


©2005  Systems  and  Software  Consortium,  Inc. 


Source:  Nidiffer  and  Dolan,  IEEE  Software,  Sept/Oct  2005 
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sals  and  Maturity  Levels 
El  by  Country 


Country 

Number  of 
Appraisals 

Maturity 
Level  1 
Reported 

Maturity 
Level  2 
Reported 

Maturity 
Level  3 
Reported 

Maturity 
Level  4 
Reported 

Maturity 
Level  5 
Reported 

Country 

Number  of 
Appraisals 

Maturity 
Level  1 
Reported 

Maturity 
Level  2 
Reported 

Maturity 
Level  3 
Reported 

Maturity 
Level  4 
Reported 

Maturity 
Level  5 
Reported 

Argentina 

^19 - 

No 

Yes 

Yes 

Yes 

Yes 

Korea,  Republic  Of78 

Yes 

Yes 

Yes 

Yes 

Yes 

Australia 

"23 

Yes 

Yes 

Yes 

Yes 

Yes 

Lat\ia 

10  or  fewer 

Austria 

10  or  fewer 

Malaysia 

"19 

No 

Yes 

Yes 

No 

Yes 

Bahrain 

10  or  fewer 

Mauritius 

10  or  fewer 

Belarus 

10  or  fewer 

Mexico 

"15 

No 

Yes 

Yes 

Yes 

Yes 

Belgium 

10  or  fewer 

Morocco 

10  or  fewer 

Brazil 

"48 

No 

Yes 

Yes 

Yes 

Yes 

Netherlands 

10  or  fewer 

Canada 

"26 

No 

Yes 

Yes 

Yes 

Yes 

New  island 

10  or  fewer 

Chile  - 

2s - ^ 

No 

Yes 

Yes 

No 

Yes 

Pakistan 

10  or  fewer 

China 

540 

Yes 

Yes 

Yes 

Yes 

Yes 

Peru 

10  or  fewer 

Colombia 

lOul  fyWer 

Philippines 

"16 

No 

Yes 

Yes 

No 

Yes 

Czech  Republic 

10  or  fewer 

Portugal 

10  or  fewer 

Denmark 

10  or  fewer 

Russia 

10  or  fewer 

Dominican  Republic  10  or  fewer 

Singapore 

10  or  fewer 

Egypt 

"17 

No 

Yes 

Yes 

Yes 

Yes 

Slovakia 

10  or  fewer 

Finland 

10  or  fewer 

South  Africa 

10  or  fewer 

France 

75 

Yes 

Yes 

Yes 

Yes 

Yes 

Spain 

"31 

No 

Yes 

Yes 

No 

Yes 

Germany 

"35 

Yes 

Yes 

Yes 

Yes 

Yes 

Sweden 

10  or  fewer 

Flong  Kong^««*  NTT 

Switzerland 

10  or  fewer 

India  C 

"204  ^ 

No 

Yes 

Yes 

Yes 

Yes 

Taiwan 

"46 

No 

Yes 

Yes 

No 

Yes 

Indonesia 

lOor'feffSr 

Thailand 

10  or  fewer 

Ireland 

10  or  fewer 

Turkey 

10  or  fewer 

Israel 

"10 

United  Kingdom 

"48 

Yes 

Yes 

Yes 

Yes 

No 

Italy 

10  or  fewer 

United  States 

718 

Yes 

Yes 

Yes 

Yes 

Yes 

Japan 

"172 

Yes 

Yes 

Yes 

Yes 

Yes 

Viet  Nam 

10  or  fewer 
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s  in  the  Digital  Spectrum  Enables 
nmunication  and  Collaboration 


Kilobit/Sec.  Megabit/Sec.  Gigabit/Sec.  Terabit/Sec. 

1,000-103  1,000,000-106  1,000,000,000-10®  1,000,000,000,000-1012 


— 

Rule  #4:  The  best  companies  are  the  best  collaborators* 


*  Friedman,  Thomas  L.  “The  World  Is  Flat,  Farrar,  Straus  and  Giroux,  2005 
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Connects  Systems... 


_ _ _  How  Future  Trends  in  Systems  and  Software  Technology 

Software  Engineering  Institute  CamegieMeUon 

©2007  Carnegie  Mellon  University 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


and  Expanded  Features 

esearcn  uenier 


3  of  Modeling  and  Simulatbn 
1  Unveils  New  Modeling  and  Simulation 


New  Aviation  Ship 
Integration  Center,  a 
state-of-the-art  research 
facility  established  in 
partnership  with  the  U.S. 
Navy  to  conduct 
modeling,  simulation, 
research,  development 
and  in-depth  analysis  for 
CVN  21 -class  aircraft 
carriers  and  other 
aviation-capable  ships. 
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Process  Improvement 


Data-Driven  (e.g.,  Six  Sigma,  Lean) 


Clarify  what  your  customer  wants  (Voice 
of  Customer) 

A  Critical  to  Quality  (CTQs) 

Determine  what  your  processes  can  do 
(Voice  of  Process) 

A  Statistical  Process  Control 

Identify  and  prioritize  improvement 
opportunities 

A  Causal  analysis  of  data 

Determine  where  your 
customers/competitors  are  going  (Voice  o1 
Business) 

^ _ 


Model-Driven  (e.g.,  CM  Ml) 


Determine  the  industry  best  practice 

A  Benchmarking,  models 

Compare  your  current  practices  to 
the  model 

A  Appraisal,  education 

Identify  and  prioritize  improvement 
opportunities 

A  Implementation 

A  Institutionalization 

Look  for  ways  to  optimize  the 
processes 

Ref.  Dr.  Rick  Hefner,  Northrop  Grumman 
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Virtual 
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Transfer 


White- 

boarding 
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lications 
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Web 

Services 


Scheduling, 
tracking, 
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applications 
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action 
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Systems 


Unstructured 
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AOreater  demands  on  systems  and  software  engineers  will  stimulate  growth  in 
the  field  -  nationally  and  internationally 

Nndustry/Gov’t  will  increasingly  focus  on  attracting,  training  and  retaining 
systems  and  software  engineering  talent  -  short  and  long  run  -  with  emphasis 
on  providing  a  Generation  Y  work  environment 

Aincreased  reliance  on  systems  and  software  engineering  processes  and 
technologies  to  effectively  manage  the  acquisition/”green”  space 

Ajhe  laws  of  Augustine’s  and  Moore  will  continue  to  hold  and  will  continue  to 
be  a  forcing  function  to  bring  the  fields  of  software  and  systems  engineering 
closer  together 

Aimprovements  in  program  risk-reduction  collaboration  mechanisms  will  be 
significant  enablers  for  increases  in  systems  and  software  engineering 
communication  and  “decision  velocity” 
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re  Engineering  Trends  That  Bode 
Joption  of  CMMI 


^Increased  need  for  a  large  number  of  complex  systems  and  systems  of 
systems  will  lead  to  investments  in  research  and  technology 

ASystems  and  software  engineers  will  continually  find  way  to  innovative  to 
reduce  complexity 

i  Increased  importance  of  modeling  and  simulation 

i  Increased  reliance  on  architectures  (top-down  and  bottoms-up) 

I  Increased  design  for  continuous  evolution  and  deployment  at  all  levels 
will  occur 

>  Understanding  users  and  their  context  will  evolve,  e.g.  leaner  system 
and  software  engineering  process  assets  on  projects 

Aincreased  customer  requests  for  system  and  software  engineering  support 
earlier  in  life  cycle 

^hift  of  systems  and  software  engineering  focus  from  the  platform  to  the 
networks 

AProcess  improvement  will  continue  to  be  important 
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>  High  Maturity  Implementation 

•  High  Maturity  Foundation 

•  Practice  Relationships 

•  Keys  to  Success 
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High  Maturity  Foundation 


So 

You  achieved  Maturity  Level  3 

Congratulations!!!!! 

And  now  youoe  ready  for  all  that  high 
maturity  stuff 


Really?????? 


^PDF 

■f  Comolete 


use  period  has  ended. 
Thank  you  for  using 


Your  compiimentary 


PDF  Compiete. 


and  Expanded  Features 


High  Maturity  Foundation 

Do  lower  Maturity  Level  PAs  look,  feel  and 
smell  differently  in  a  High  Maturity 
Organization???? 

YOU  BETCHA!!!!!!!!!!!!!!!!!!! 

They  serve  as  the  foundation  for  ML4  and  MLS 
practices 
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High  Maturity  Foundation 


Maturity  Level  5  PAs  -  A  Qualitative  Summary 

CAR  V  If  something  is  wrong,  or  needs  to  be  better,  get 
the  right  people  together,  detetmine  the  real  problem, 
and  fix  it. 

OID  T  Try  to  get  better  T  especially  in  the  areas  that  are 
most  important.  Be  pro-active  in  looking  for  ways  to 
get  better  in  these  important  areas. 


I’ll  bet  you’re  already  doing  this!!!! 
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High  Maturity  Foundation 


PPQA  T  Are  you  performing  trend  analysis  on  non- 
compliance  items? 

PMC  T  Are  you  determining  the  real  cause  of 
deviations  from  plans? 

VER  &  VAL  V  Are  you  performing  trend  analysis  on 
issues  arising  from  Peer  Reviews? 

Are  you  performing  trend  analysis  and 
determining  the  rea  cause  of  problems  found  in 
T&E? 
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High  Maturity  Foundation 


OPF  T  How  pro-active  is  your  PI  program? 
How  do  you  prioritize  PI  initiatives?  How  do 
you  know  if  improvements  are  really 
improvements? 

MA  V  What  is  the  basis  for  those  objectives? 
Do  your  measures  really  tie  to  the 
objectives?  Are  your  ofSerational  definitions 
sound? 
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High  Maturity  Foundation 

OPD  I  Do  you  truly  have  a  set  of  standard 
processes?  Are  the  process  elements  well 
defined? 

GP  3.2  V  Are  you  really  collecting  improvement 
information?  Is  it  quantified? 

How  do  you  know  if  things  are  going  well? 
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High  Maturity  Foundation 


OPP  QPM 
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High  Maturity  Foundation 


How  do  you  establish  Quality  and  Performance 
Baselines  and  Models  without  the  data  from 
QPM? 


How  do  you  establish  the  framework  for  QPM 
without  QPP? 


See  High  Maturity  Foundation 
(I  vote  for  the  chicken) 
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Practice  Relationships 


OPP  SP  1.3  Establish  quality  and 


process  performance  objectives 


•  QPM  SP  1 .1  Establish  the  project®  objectives 

•  OID  SP  1 .1  Collect  and  analyze  improvement 
proposals 

•  OID  SP  1 .2  Identify  and  analyze  innovations 

•  OID  SP  1 .4  Select  improvements  for 
deployment 
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Practice  Relationships 

OPP  SP  1.4  Establish  process-performance  baselines 
OPP  SP  1.5  Establish  process-performance  models 

•  QPM  SP  1 .2  Compose  the  defined  process 

(and  most  of  QPM) 

•  OID  SP  1 .1  Collect  and  analyze  improvement 
proposals 

•  OID  SP  1 .2  Identify  and  analyze  innovations 

•  OID  SP  2.3  Measure  improvement  effects 
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Practice  Relationships 

QPMSP1.1  Establish  the  project’s 
objectives 

QPMSP1.4  Manage  project 
performance 

QPM  SP  2.3  Monitor  performance  of 
the  selected  subprocesses 

CAR  SP  1 .1  Select  data  for  analysis 
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Practice  Relationships 

QPM  SP  1.2  Compose  the  defined  process 

QPM  SP  1 .4  Manage  project  performance 

QPM  SP  2.3  Monitor  performance  of  the 
seiected  subprocesses 

CAR  SP  2.1  Implement  the  action  proposals 
CAR  SP  2.2  Evaluate  the  effect  of  changes 
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Common  Misconceptions: 

Processes  vs  Subprocesses 

Subprocess  T  a  defined  component  of  a  larger  defined  process 
that  may  be  decomposed  further 

ML4  statistical  management  is  at  this  ievei 
Process-performance  modeis 

The  use  of  product  and/or  process  measurements  coiiected  in 
one  activity  to  predict  the  resuits  of  another  activity 

Exampie  V  Defects  found  in  a  requirements  Peer  Review  used  to 
determine  the  number  of  defects  that  wiil  be  found  in  integration 
testing 
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Keys  to  Success 


Be  sure  of  your  foundation 
Keep  it  practical  T  not  academic 
Use  the  informative  material 

AmL4  =  special  cause  variation 
A  MLS  =  common  cause  variation 


•  Treat  the  4  PAs  as  one 

•  Dond  be  overly  concerned  with  Project  vs  Org  with 
CAR  and  01 D  activities 

•  Use  qualified  people  and  tools  to  develop  process- 
performance  models 
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Fallacy.  Any  time  spent  on  the  higher  maturity  level  practices  while 
attempting  to  achieve  CMMI  ML2  or  MLS  is,  by  definition,  wasted  effort. 

Radical  Thought  #1\  Any  time  spent  implementing  policies  and  practices 
at  ML2  and  MLS  that  does  not  support  the  higher  maturity  level  CMMI 
practices  violates  the  intent  of  the  model. 

A  Otherwise  serious  rework  can  be  required  to  achieve  ML4  and  MLS. 

A  At  the  extreme,  ML2  and  MLS  practices  are  implemented  poorly  and  for  all 
the  wrong  reasons. 

Radical  Thought  #2:  You  need  to  understand  ML4  and  MLS  concepts 
before  you  can  properly  interpret  ML2  and  MLS  for  your  organization. 
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The  phrase  fprocess  improvementoimplies  improving  the  performance  of 
a  given  process  or  set  of  processes  with  respect  to  some  objective 
standard. 

A  CMMI  does  not  specify  performance  standards,  it  only  implies  their 
existence. 

Improving  performance  with  respect  to  an  objective  standard  implies  that 
something  about  the  process  will  be  measured. 

"If  you  can  not  measure  it,  you  can  not  improve  it."  T  Lord  Kelvin 
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s  Performance? 


Process  Performance:  nA  measure  of  actual  results  achieved  by 
following  a  process,  it  is  characterized  by  both  process  measures  (e.g., 
effort,  cycie  time,  and  defect  removai  efficiency)  and  product  measures 
(e.g.,  reiiabiiity,  defect  density,  and  response  time). 6 

Process  Performance  Baseline  [PPB]:  documented  characterization 

of  the  actual  results  achieved  by  following  a  process,  which  is  used  as  a 
benchmark  for  comparing  actual  process  performance  against  expected 
process  performance. 6 

Process  Performance  Model  [PPM]:  description  of  the  relationships 

among  attributes  of  a  process  and  its  work  products  that  is  developed  from 
historical  process-performance  data  and  calibrated  using  collected 
process  and  product  measures  from  the  project  and  that  is  used  to  predict 
results  to  be  achieved  by  following  a  process. 6 
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A  process  performance  model  is  used  for  essential  process  improvement 
activities. 

A  explain  past  performance  (e.g.  the  PPBs) 

A  predict  future  performance  (may  look  like  the  PPBs  in  part) 

A  indicate  what  (else)  to  measure 
A  identify  opportunities  for  improvement 
Are  these  purposes  guiding  your  ML2  and  MLS  practices? 

Can  you  do  these  things  without  the  statistical  rigor  demanded  by  ML4/5? 
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me 


In  Mythical  Man-Month,  Fred  Brooks  gave  this  gross  characterization  of 
effort  distribution  of  programming  processes. 


Planning  and  design 

1/3 

Coding 

1/6 

Unit  Test 

1/4 

System  Test 

1/4 

This  characterization  provides  a  baseline  (although  not  a  complete  PPB 
in  the  CMMI  sense)  for  process  performance  at  IBM  in  the  late  1960s. 

It  can  help  to  explain  past  process  performance,  and  when  combined 
with  an  estimate  of  effort  on  a  future  similar  project,  it  can  help  to  predict 
future  performance  of  the  process  (although  not  yet  a  PPM). 
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nd  What’s  Missing 


Based  on  this  historical  benchmark,  if  I  have  an  enhancement  project  that 
I  estimate  at  100  hours,  my  predicted  performance  would  be: 


Planning  and  design 

33  hrs. 

Coding 

17hrs. 

Unit  Test 

25  hrs. 

System  Test 

25  hrs. 

Do  I  have  any  idea  of  how  relevant  this  prediction  is  to  me? 

Do  I  have  any  idea  of  which  activities  have  the  most  opportunity  for 
improvement? 

Do  I  have  any  idea  of  how  to  push  this  in  the  direction  of  a  true  PPM? 
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Project  Data 


Planning  and  Design  27  hrs. 

Coding  38  hrs.  (untii  clean  compile) 

Unit  Test  38  hrs.  (21  defects  found) 

System  Test  35  hrs.  (1 1  defects  found,  3  passes  of  test  suite) 

Totai  138  hrs.  (vs.  estimated  100  hrs.) 

Do  i  have  any  idea  of  how  mormaiothis  may  or  may  not  be  for  me? 

What  shouid  I  be  measuring  in  more  detaii  on  future  projects? 
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s  I  Want  to  Know 


How  much  time  in  planning  vs.  design  vs.  understanding 
requirements? 

How  much  time  fixing  compile/environmental  defects? 

How  many  unit  test  cases?  How  many  passes  (partial  and 
complete)? 

How  much  time  executing  system  test  suite  vs.  fixing  defects? 
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Process  problems: 

1.  No  way  to  tell  from  the  gathered  data  how  much  time  was  spent  on 
planning  vs.  design  or  other  activities  In  that  phase 

2.  No  way  to  tell  how  much  time  in  coding  was  in  fixing  compile  or  link 
defects 

3.  How  much  test  time  in  testing  vs.  fixing 
Proposed  solutions: 

1.  Tag  all  hours  with  (planningo(&esignQ(analysiso®ther6 

2.  Tag  all  hours  in  coding  with  fflodeoor  ©ompile/fixo 

3.  Tag  all  hours  In  test  with  (testing  <case  #>6or  (defect  find/fix  <bug  #>6 
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After  a  couple  more  similar  projects: 


Phase 

Project  1 

Project  2 

Project  3 

Cum.  % 

Planning  &  design 

27 

38 

47 

25.8 

Code 

38 

26 

39 

23.5 

Unit  Test 

38 

28 

43 

25.2 

System  Test 

35 

29 

47 

25.5 

Totals  I  Act./Est. 

138/100 

122/110 

175/120 

100.0 

From  hour  tags,  understanding  requirements  is  about  1/2  of  fplanning 
and  designotime,  actual  planning  and  design  about  1/4  each. 

Defect  fix  times  in  UT  and  ST  are  about  70-80%  of  the  total  test  time, 
more  if  you  count  all  of  the  extra  passes  needed  in  the  test  suites. 
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On  average,  the  phases  are  fairly  equally  balanced. 

However,  looking  at  individual  efforts  for  fplanning  and  designoas  a 
predictor,  those  were  much  different  (about  1/5, 1/3, 1/4,  respectively). 

Prediction  of  the  cumulative  time  measured  after  ffcodingoseems  much 
more  reliable,  always  about  1/2  the  total  project  time  in  (planning  &  designo 
and  ncodingq  the  other  half  in  fiinit  testoand  reystem  testo 

fUnit  testois  a  fairly  good  predictor  of  ffeystem  testq  even  though  the  tests 
run  are  completely  different. 

Estimates  arenftvery  good  (about  30%  average  overrun).  If  only  we  didnft 
have  to  do  reystem  testae 

Characterizing  key  relationships  and  their  variation  statistically  helps  to 
make  a  PPM. 
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Process  problems: 

1.  Many  defects  and  multiple  test  suite  passes  (waste  of  time!)  are  due  to 
not  being  able  to  find  all  defects  in  the  first  pass. 

2.  Effort  overruns  are  creating  increased  project  tracking  overhead  (i.e. 
management  pressure). 

Proposed  solutions: 

1.  Provide  inspection  training  &  require  inspections  of  all  code;  log  all 
inspection  effort  and  defect  data. 

2.  Increase  effort  estimates  by  an  amount  large  enough  to  allow  for 
variation  in  key  performance  indicators. 
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Phase 

Project  4 

Project  5 

Project  6 

Cum  % 

Planning  &  design 

20 

40 

40 

22.9 

Coding 

25 

45 

75 

33.2 

Unit  Test 

21 

33 

37 

22.2 

System  Test 

24 

30 

32 

21.7 

Totals  I  Act./Est. 

95/75 

148/185 

184/215 

100.0 

Inspection  time  was  rolied  into  ncodingosince  it  is  the  code  being 
inspected,  about  1/4  of  the  totai  hcodingo  effort. 

UT  and  ST  about  42%  of  totai  effort,  down  from  about  51%. 

Actuais  are  about  1 1  %  under  estimates  on  average,  but  they  wouid  have 
been  about  18-19%  over  if  not  for  1/3  ffeffort  adjustmento 
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ita  Analysis  /  More  Questions? 


On  this  basis  (18-19%  vs.  30+%  over),  inspections  seem  to 
be  working.  (Remember  to  compare  apples  to  apples!) 

Defects  found  in  code  inspection  tend  to  be  simple  coding 
errors,  with  the  occasional  design  defect. 

About  60%  of  total  testing  effort  still  devoted  to  finding  defects 
and  multiple  test  suite  runs.  A  majority  of  defects  now  seem 
to  be  design  issues  (used  to  be  about  even  between  design 
and  coding  issues). 
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rPlanning  and  designoand  rtodingoeffort  seem  to  relate 
directly  to  the  scope  of  the  project. 

E1  (effort  before  test)  =  f(scope) 

While  loosely  related  to  scope,  testing  effort  seems  more 
directly  related  to  the  number  of  defects  and  the  number  of 
test  suite  passes. 

E2  (effort  in  test)  =  f(defects)  +  (effort  in  1  pass  through  UT  and  ST)* 


*  -  probably  related  to  scope! 
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Problem  description: 

1.  Effort  estimates  under  management  pressure 

2.  Still  lots  of  fwastedotime  in  UT  and  ST 
Process  proposal: 

1.  Reduce  the  1/3  reffort  adjustmentoto  1/5 

2.  Create  more  nnspectableodesigns  by  using  design  templates  or 
architectural  views;  inspect  for  common  design  defects  found  in  test 

Note:  Is  either  proposal  f%tatistically  soundcP  (Probably  not.) 

What  would  you  do  instead?  (Hmmme  e  .) 
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Rhetorical  Questions 


Is  the  gathering  and  use  of  data  by  the  people  doing  the  job 
high-  or  low-maturity? 

Do  I  have  to  be  ML4  or  MLS  to  do  any  of  this? 

Will  this  make  you  ML4  or  MLS  (or  any  level)  if  you  do  this? 

Have  you  seen  control  charts?  Complex  mathematical 
models? 

Do  you  think  that  such  practices  would  help  speed  you  on 
your  way  to  ML4/S? 
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The  Voice  of  the  Business  (your  boss)  tells  you  that  your 
performance  goal  for  next  year  is  to  deliver  your  projects  in  85%  of 
the  calendar  time  that  you  estimate  with  fewer  defects  delivered  to 
the  external  customer. 

Your  standard  process  simply  cannot  perform  to  this  level. 

There  are  two  basic  types  of  response  to  such  pressure. 

A  low  maturity  (try  harder!  i.e.  more  than  40  hours/week) 

A  high  maturity  (work  smarter!) 
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Your  current  process  baseline  (still  not  a  PPB)  looks  like  this: 


Phase 

%  actual  effort 

Defect  yield* 

Notes 

Planning  and  design 

30 

40% 

Based  on  defects 
reported  from  the  field. 

Early  yields  are  from 
inspections. 

Single  pass  of  UT  and 
ST  -10%  of  effort. 

Coding 

35 

50% 

Unit  Test 

15 

40% 

System  Test 

20 

40% 

You  need  to  squeeze  15%  out  of  your  average  estimated  lifecycle  effort. 

You  are  still  doing  multiple  passes  of  extensive  (and  expensive)  testing. 

If  only  you  could  reduce  the  number  of  passes  in  LIT  and  STe 
*  Defect  yield  i  percentage  of  defects  found  in  phase  that  were  present  or  injected  in  that  phase 
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If  you  could  increase  yields  in  the  eariy  phases,  you  couid  further  reduce 
the  number  of  defects  in  LIT  and  ST  and,  more  significantly,  finally  reduce 
the  number  of  test  passes. 

You  canftwave  a  magic  wand  at  inspections  and  say,  fFind  more  defectsio 

But  youwe  heard  or  read  of  other  methods  that  drastically  reduce  the 
numbers  of  test  defects. 

A  PSP/TSP 

A  Correctness  by  Construction 
A  Test-Driven  Development 

Pick  one.  Investigate.  Better  yet,  get  your  process  group  to  do  it!  Or  at 
least  pay  for  the  training.  (But  donfttell  them  it®  the  MLS  thing  to  do,  it 
might  scare  theme  ) 
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Process  is  like  exercise. 

If  you  arend  used  to  it,  it  hurts. 

Once  you  do  get  used  to  it,  if  it  still  hurts,  you  are  either 

A  trying  to  do  too  much 
A  doing  it  wrong 

It  gives  you  more  time  and  energy  to  do  all  the  other  stuff  you 
know  you  ought  to  be  doing,  so  you  get  more  done. 

It®  usually  a  little  easier  and  a  lot  more  fun  when  performed  in 
groups. 


Software  Engineering  Institute  CarnegieMellon 


Thought  Before  Action 

James  D.  McHale,  CMMI  NDIA2007 


©2007  Carnegie  Mellon  University 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


P 


member 


CMMI  is  a  model  that  encourages  (and  ultimately  demands) 
process  performance  improvement. 

While  it  wondget  you  a  ML4/ML5  rating,  you  can  begin 
implementation  of  high-maturity  concepts  with  very  simple 
models  and  techniques.  (Let  the  data  show  the  way!) 

Significantly  improved  performance  on  your  projects  is 
achievable  now,  regardless  of  maturity  level. 
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Knowing  your  goals 

Collecting  data 

When  do  you  have  a 
baseline 

What  is  a  model 
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■  Intended  for  people  who  are  new  to  baselines  and  models. 


■  Uses  an  example  that  everyone  can  relate  to,...  how  much  time 
should  I  allocate  to  get  to  the  airport  gate  on  time. 

■  If  you  understand  basic  principles,  you  can  apply  it  to  your  work. 

■  The  bottom  left  corner  of  each  slide  describes  how  the  same 
principles  can  be  applied  to  peer  reviews. 
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NORTHROP  GRUMMAN 


oals,  i.e..  What  is  Important  to  You? 


Goal  1 


Goal  2 


(focus  for  this  presentation) 


possible  so  I  can 
spend  more  time 
at  home  instead  of 
sitting  in  an  airport. 


Save  money. 
There  are  different 
ways  to  get  to  the 
airport  that  have 
different  costs. 


Never  create  a  baseline  and  model  if  you  have  no  goal. 

Peer  Reviews:  Typical  peer  review  goals  are  to  find  more  defects  and  to  be  more  efficient. 
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Peer  Reviews:  Pain  is  the  number  of  defects  found  during  integration  and  test  and  system  test. 
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Use  a  Stable  Process 


Peer  Reviews:  At  first,  you  might  have  the  number  of  defects  over  the  project  life  cycle  so  the  process  may  be  unstable. 
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3ally  Unstable? 


Peer  Reviews:  Breaking  down  the  data  by  life-cycle  phase,  i.e.,  requirements,  design,  code,  test,  etc.  may  show  the  data  is  not  unstable. 
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Taking  the  Train 


NORTHROP  GRUMMAN 


Disaggregate  to  see  if  there  is  a  reason  for  the  5  outliers. 


Peer  Reviews:  You  might  do  a  control  chart  of  just  code  reviews  and  you  might  get  some  outliers. 
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Dns  is  Rush  Hour 


No 


Yes 


Peer  Reviews:  An  outlier  could  be  complex  code,  an  inexperienced  developer,  an  unusually  large  number  of  reviewers,  etc. 
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But  is  the  difference  significant  enough  to  have  two  baselines? 


Peer  Reviews:  Defects  may  be  different  for  inexperienced  developers  but  it  may  not  be  significant,  whereas  #  of  reviewers  might  be. 
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riance  Provides  the  Answer 


95%  Confidence  Intervals  for  Sigmas 

Factor  Levels 

* - • - * 

No 

• - • 

Yes 

I  I 

10 

F-Test 

Test  Statistic:  16.452 


20 


30 


P-Vaiue 


0.00 


Bo^plots 


Test  for  Equal  Variance  shows  the 
difference  is  statistically  significant. 
P-Value  <  0.05  is  significant. 

Similar  analysis  should  be  done  for  other 
process  variations,  not  just  rush  hour. 


The  difference  is  significant  enough  to  warrant  two  baselines, 


Peer  Reviews:  One  area  that  has  a  significant  difference  is  whether  people  review  thoroughly  before  the  meeting. 
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Rush  Hour  Baseline 


Not  Rush  Hour  Baseline 


Mean 

St  Dev 

Variance 

Skewness 

Kurtosis 

N 


55.4286 

3.8668 

14.9524 

-2.19901 

5.49992 

7 


Minimum 
1st  Quartile 
Median 
3rd  Quartile 
Maximum 


47.0000 

56.0000 

56.0000 

57.0000 

59.0000 


13] 


The  standard 
deviation  is  high. 
Consider  lower- 
ievei  baseiines  to 
refine  this 
baseiine. 


Note:  Collect  at 
least  15  points. 


□  s 

Mean 

^tDev 

Variance 

Ske'iA/ness 

Kurtosis 

N 


ID 

67.0000 

15.6844 

246 

1.28677 

2.89383 

13 


Minimum  43.000 

1st  Quartile  57.000 

Median  64.000 

3rd  Quartile  75.500 

Maximum  107.000 


Baselines  provide  a  range  and  distribution  for  performance. 


Peer  Reviews:  The  project  may  have  separate  baselines  for  peer  reviews  done  with  and  without  customers  and  managers. 
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Subprocess 

Step 


Actual  data  collection  form 
used  during  train  trips.  Data 
was  collected  over  the  last  2.5 
years  for  17  trips. 


0 

Home  to  Train 

Time  leave  4^u^e 

© 

Waiting  for  Train 

Time  sit  on  bench  at  train  station 

0 

Train  Ride 

Time  train  leaves 

0 

Waiting  for  Shuttle 

Time  train  stops  at  Aviation 

© 

Shuttle  Ride 

Time  shuttle  leaves 

© 

Terminal  1  to  Terminal  6 

Time  shuttle  at  Terminal  1 

© 

Terminal  6  Door  to  Gate 

Time  at  Door  C  at  Terminal  6 

Time  at  United  gate 

Break  down  the  process  into  measurable  subprocesses. 


Peer  Reviews:  Subprocesses  for  peer  reviews  include  preparing,  reviewing  before  the  meeting,  the  meeting,  closing  action  items. 
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3cesses  for  Process  Variation  (i  of  2) 


15  - 


ID  - 


5  - 


55^ 


SH  - 


TS  - 


TH  - 


Home  to  Train 
(6  to  17  min) 


Train  Ride 
(7  to  8  min) 


The  distribution  / 
range  is 

unacceptable  for 
1  and  4.  Need  to 
understand 
better. 


Train  Ride  is 
extremely 
stable.  No 
improvements 
possible. 


Waiting  for  Train 
(1  to  10  min) 


ID  - 


Waiting  for  Shuttle 
(2  to  15  min) 


ID  - 


D  - 


If  youoe  unhappy  with  the  range,  investigate  for  improvements, 


Peer  Reviews:  Probably  see  variation  depending  on  the  number  of  reviewers  and  the  size  of  the  product  being  reviewed. 
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3cesses  for  Process  Variation  (2  of  2) 


Shuttle  Ride 
(6  to  18  min) 


Terminal  6  Door  to  Gate 
(4  to  50  min) 


Obvious  the 
process  was 
improved. 
Discovered  it  is 
faster  to  get  off  the 
shuttie  and  walk  to 
Terminal  6. 


5  and  7  appear 
stable  but  both 
have  outliers  to 
investigate 


Terminal  1  to  Terminal  6 
(6  to  12  min) 


Outliers  are  special  causes  which  should  be  investigated. 


Peer  Reviews:  Variation  in  preparing  for  a  meeting  could  be  whether  the  customer  is  there,  in  which  case  briefings  are  created. 
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Home  to  Train 

yjv - 


ed  Reasons  for  Process  Variation 


IS  - 


ID  - 


S  - 


^^1 


Driving  vs 
Walking 


Shuttle  vs 
Walking 


Terminal  1  to 

(e)^ - 


Terminal  6 


1D  - 


S  - 


1D 


15 


"T" 

ID 


—r 

15 


0 


Shuttle  Ride 

No 


a]  - 


ID  - 


D  - 


"T" 

ID 


Rush  Hour  vs 
No  Rush  Hour 


Normal  vs 
Special  Case 


erminal  6  Door  to  Gate 

_ 


Termi 

=0^ 


4D  - 


□  - 


Went  to  wrong  terminal, 
flight  cancelled,  Premier 
security  disappeared 


ID 


15 


Recall  from  previous  slides  about  significant  differences. 


Peer  Reviews:  Collect  enough  data  from  each  peer  review  subprocess  so  the  graphs  clearly  show  the  difference. 
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ed  3  Variables  for  the  Model 


Rush  Hour 

(No  translation 
needed) 


Raining 

(Translated  for 
walking  to  the  train 
and  Terminal  6) 


Normal 

Situation 

(Translated  for 
special  causes) 


Use  terms  that  users  of  the  model  will  understand. 


Peer  Reviews:  This  is  not  really  a  problem  for  peer  reviews,  except  maybe  hnexperienced  developerowhich  may  be  sensitive. 
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Monte 


Minimum 

Median 

Maximum 

Standard  Carlo  Data 

Minutes 

Minutes 

Minutes 

Mean 

Deviation  Type 

Train  Details  for  Monte  Carlo  Simulation 

Home  to  Train  (Car) 

6 

7 

8 

Triangular 

Home  to  Train  (Walk) 

14.60 

1.24  Lognormal 

Wait  for  Train 

1 

3 

5 

Triangular 

Train  Ride 

3 

8 

9 

Triangular 

Wait  for  Shuttle  (Rush  Hour) 

3 

4 

7 

Triangular 

Wait  for  Shuttle  (No  Rush  Hour) 

2 

8 

16 

Triangula!^  — 

Shuttle  Ride  (Rush  Hour) 

6 

11 

18 

Triangular^ 

Shuttle  Ride  (No  Rush  Hour) 

8 

8 

10 

Triangular 

Terminal  1  to  United  (Shuttle) 

9 

10 

12 

Triangular 

Terminal  1  to  United  (Walk) 

6 

7 

8 

Triangular 

To  Gate  (Special  Cause) 

14 

30 

50 

Triangular 

To  Gate  (Normal) 

9.39 

3.10  Normal 

Variables  for  the  Simulation 

Rush  Hour? 
Raining? 

Normal  Situation? 


3  variables  for 
the  simulation 


Need  to  know 
whether  Triangular, 
Normal,  Lognormal, 
or  a  constant 
should  be  used  for 
the  simulation 


Simulations  assume  you  understand  your  data. 


Peer  Reviews:  Can  simulate  the  estimated  number  of  defects,  the  estimated  hours  for  doing  peer  reviews,  etc. 
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^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


NORTHROP  GRUMMAN 


Train  Trips 


This  is  the  actual  model  1  use 
when  1  take  the  train.  Based 
on  the  baselines,  it  says 
what  time  to  leave  the  house. 

Enter  Departure  Time: 

8:20  AM 

Look  at  Minimum  Train  Time: 

6:55  AM 

Enter  Best  Train  Departure  Time: 

6:53  AM 

Rush  Hour  (Yes  or  No): 

Yes 

Walk  to  Train  (Yes  or  No): 

Yes 

Walk  to  Door  C  (Yes  or  No): 

Yes 

When  to  Leave  House 

6:36  AM 

Time  Home  to  Bench 

0:14 

0:07 

6:36  AM 

Time  Waiting  for  Train  to  Leave 

0:03 

6:50  AM 

6:53  AM 

Minimum  Train  Departure  Time 

6:55  AM 

Time  Train  Start  to  Stop 

0:08 

7:03  AM 

Time  Waiting  for  Shuttle  to  Leave 

0:10 

0:05 

7:11  AM 

Time  Shuttle  Drive  to  Terminal  1 

0:11 

0:09 

7:16  AM 

Time  Terminal  1  to  Door  C 

0:11 

0:07 

7:25  AM 

Time  Door  C  to  Gate 

0:13 

7:32  AM 

Time  Bathroom  Si  Stand  By  Gate 

0:05 

7:45  AM 

Time  Sitting  on  Plane 

0:30 

7:50  AM 

8:20  AM 

Monte  Carlo  Simulation  Output 

(Note:  This  is  not  the  actual  data  for  the  train) 


1,000  Trials 


Frequency  View 


000  Displayed 


Estimated  PCJR  Hours 


fd 


0.05 


0.04 


0.03 


0.02 


0.01 


0.00 


20.00 


40.00 


60.00 


80.00 


100.00 


120.00 


Certainty:  jSO.Od  '  % 


<3  j91.51 


Select  a  percent.  Means 
80%  probability  it  will 
take  <=  91.51  minutes. 


Models  are  powerful  for  predicting/estimating  the  future. 


Peer  Reviews:  Probably  don^need  the  one  on  the  left,  but  doing  a  Monte  Carlo  simulation  on  the  right  would  be  useful. 
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^  Complete 
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PDF  Complete. 
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NmrrHROP  Grumman 


■  Identify  your  goals  before  creating  any  baselines  and 
models 

■  Analyze  and  disaggregate  the  data  until  it  is  stable 
(no  special  causes) 

■  Create  multiple  baselines  when  process  variation 
(rush  hour)  is  significant 

■  Understand  each  subprocess  thoroughly  to  create  better 
models.  Analyzing  subprocesses  uncovers  process 
variables  (rush  hour,  car  vs  walking,  shuttle  vs  walking, 
flight  problems,  etc.) 

■  Create  models  to  estimate  /  predict  the  future 


Diane.Mizukami@ngc.com,  31 0-921  -1 939 
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^  ^  F  I  N I N  e  THE  FUTURE 

Expanding 
Statistical  Process 
Control  Across  All 
Engineering 
Disciplines 

A  Sequence  of  Practical  Case  Studies 


November  15,  2007 

Richard  L.  W.  Welch,  PhD 

Northrop  Grumman  Corporation 
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^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

anct^xpanded  Features 


Purpose 

■  What  you  will  see 

SPC  principles 
Prior  presentations 

■  2005  I  Log-cost  model  for  controlling  software  code 
inspections 

■  2006  I  Statistical  Process  Control  early  in  the 
system/software  life  cycle 

Case  studies  from  other  disciplines 

■  Test 

■  Avionics 

■  Vehicle 

■  Logistics 

Summary 

MOHTHRaP  CRUMMAR! 


ISER-MLB-PR-07-146 
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^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


/rac/e  to 

'  anct^xpanded  Features 


Illustrate  a  variety  of  statistical  process  control  (SPC) 
applications  with  realistic  engineering  case  studies 

■  Multiple  engineering  disciplines 

■  Software,  hardware,  logistics 

■  Process  improvements  applied  to  selected  processes 
when  it  makes  sense  for  the  business 

Portray  operations  of  a  large  organization  that  has 
been  at  Level  5  for  21/2  years 

Suggest  a  potential  range  of  SPC  applications  beyond 
software 
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^  Complete 


Your  complimentary 
use  period  has  ended. 
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PDF  Complete. 
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^ill 


Lots  of  control  charts 

But  that’s  not  the  point  -  you  should  focus  on 

■  Broad  applicability  of  SPC  techniques  to  all  engineering 
disciplines 

■  Major  business  themes  that  emerge 

■  Cost 

■  Schedule 

■  Quality 

■  Vast  majority  of  optimizing  process  improvements  are 
simple  in  nature 

■  But  so  is  rocket  science,  that’s  why  it  works 

Occasional  out-of-control  points 

■  All  examples  were  taken  from  fliveoproject  data 

■  Special  causes  of  variation  do  occur,  that®  why  we 
use  SPC  to  manage  projects 
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PDF  Complete. 
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pies 


Listening  to  the  Voice  of  the  Process 


Upper  Control  Limit 


Average  performance 


Lower  Control  Limit 


A  stable  process 

operates  within  the 
control  limits 
99.7%  of  the  time 


Analysis  of 

■  Special  cause  variation  focuses  on  recognizing  & 
preventing  deviations  from  this  pattern 

■  Offers  superior  project  management  results 

■  Common  cause  variation  focuses  on  improving  the 

average  and  tightening  the  control  limits 

■  Offers  opportunities  for  systematic  process  improvement  that 
company  &  industry  benchmarks  indicate  yields  a  return  on 
investment  averaging  between  4:1  &  6:1 
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^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


ection 


Statistical  control  is  imposed  on  sub-processes  at  an 
elemental  level  in  the  process  architecture 

Processes  are  selected  based  on  their 

■  Statistical  suitability  T  mecessary  conditionso 

■  Business  significance  T  fsufficient  conditionso 

Business  checklist 

■  Is  the  candidate  sub-process  a  component  of  a  project® 
defined  fkeyoprocess? 

■  Is  it  significant  to  success  of  a  business  pian  goai? 

■  Is  it  a  significant  contributor  to  an  important  estimating  metric  in 
the  discipline? 

■  Is  there  an  identified  business  need  for  predictable 
performance  as  projects  execute  the  subprocess? 

■  Cost,  schedule  or  quality 

■  Is  there  risk  if  subprocess  variation  is  not  understood  or 
controlled? 
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Complete 
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PDF  Complete. 
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ntations 


it 


2005  -  Author  demonstrated 
applicability  of  a  log-cost 
model  to  control  software  code 
inspections 


■  2006  -  Author  demonstrated 
how  to  use  the  log-cost  model 
to  control  peer  reviews  early  in 
the  system/software  life  cycle 


■  fDutstanding  Presentation  for 
High  Maturityo 

■  rConference  Winner© 


NORTHROP  GRUMMAN 

TeMNING  the  FUTURE 


Logarithms 
Can  Be  Your 
Friends 


Controlling  Peer  Review  Costs 


"(ic  November  16,  2005 

Richard  L.  W.  Welch,  PhD 

Chief  Statistician 

Northrop  Grumman  Corporation 


NORTHROP  GRUMMAN 


Statistical  Control 
ys  of  System  and 
''  Software  Design 
Activities 


n3  November  15,  2006 


Richard  L.  W.  Welch,  PhD 

Chief  Statistician 
April  King 

Systems  Engineer 

Northrop  Grumman  Corporation 


Note:  Prior  CMMI  Technology  Conference  &  User  Group  papers  are 
published  on-line  at:  http://www.dtic.mil/ndia/ 
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Managed  Processes 


Covered  in  the  Prior  Presentations 


System  Engineering 

■  System  design  &  system 
architecture  peer  reviews  of 

■  System  threads 

■  System  model  (structure 
diagrams) 

■  Physical  model 

■  UML  diagrams 

■  System  &  software 
requirements  peer  reviews  of 

■  Proposed  specification 
changes 


Software  Engineering 

■  Software  design  peer  reviews  of 

■  Software  threads 

■  Physical  model 

■  Component/task  descriptions 

■  Data  model 

■  Software  code  inspections 

Test  &  Engineering 

■  Peer  reviews  of  test  plans, 
procedures  &  reports 
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^  Complete 
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use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 
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Managed  Processes 


other  Engineering  Baselines 


System  Engineering 

■  System  product  errors 
Software  Engineering 

■  Software  build  process 

■  Software  build  returns 

■  Software  test  returns 
Test  &  Engineering 

■  System  Integration  Lab  (SIL) 
scheduling 

■  Flight  Test  Card  development 

Vehicie  Engineering 

■  Electro-mechanical  drawing 
errors 

■  Vehicle  subsystems  (i.e.,  crew 
&  equipment)  drawing  errors 


Avionics 

■  Discrepancy  Inspection  Report 
(DIR)  processing 

■  Avionics  Drawing  Sign-off 

■  Field  Service  Engineering 
Request  (FSER)  processing 

■  Management  of  seller  issues 

Logistics 

■  Air  Force  Tech  Order  (AFTO) 
processing  of  the 

■  Total  contractor  schedule 

■  LSA  group  schedule 

■  Integrated  electronic  technical 
manual  (lETM)  delivered  quality 
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Baselines  span  all  life  cycle  phases  & 


disciplines 


PiOKTHROR  GRUMMAPi 


Note:  baselines  highlighted  in  italics  are  featured  in  this  presentation. 
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An  early  2005  analysis  determined  that  improved 
System  integration  Lab  (SiL)  resource  utiiization  could 
provide  significant  cost  savings 

■  Scheduled  shifts  not  worked  waste  Lab  Ops  resources 

■  Unplanned,  late  requests  for  lab  support  induce 
overtime  expenses 

Statistical  analysis  of  past  year’s  data  revealed  the 
process  was  stable  (with  two  unusual  exceptions) 


■85*  BO*  -75*  -70* 


■55*  -50*  -45* 


SIL  Utilization  Planning  Performance 

OCTL  Integration  Lab 


/\  A 

AL  ak\ 

^  V\ 

Hurricanes 

.V# 


Week  Ending  (Date) 

Positive  =  Worked  more  Shifts  than  Planned 


Negative  =  Worked  less  Shifts  than  Planned 


Worksheet:  MINITAB.MTW;  2/18/2005 
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:ion  Scheduling 


Process  Title 


Lab  Utilization  Scheduling 


Process 

Overview 


Process  Definition 


various  projects. 


Provides  deconflicted  and  effective  Lab  utilization  by 


Input 


Varied  Integrated  Product 
Team  (IPT)  Requirements 


Sub-Processes  Steps 


AIPT  Rep  Identifies  Requirements 
ANext  Months  Baseline  Established 
AWeekly  Schedules  Developed  &  Posted 
AWeekly  Schedules  Marked  to  Reflect 
Actuals 

APIanned  (Monthly  Baseline)  Versus  Actual 
Metric  Created 


Output 


ALong  Range  Schedule 
ANext  Month  Baseline 
AWeekly  Schedules 
A  Planned  Versus  Actual 
Metric 


Applicable  Procedures 


ALong  Range  Lab  Utilization  Scheduling 
AWeekly  Lab  Scheduling 
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nt  Focus 


Training 

■  Re-affirmed  the  need  for  accurate  planning 

■  Revised  lab  planning  procedures  disseminated  widely 

Toois 

■  Planned  vs.  Actual  utilization  spreadsheet  T  tracks  the 
lab  utilization  deviations 

Process 

■  Steering  Committee  approval  of  remedial  actions 

■  Integrated  Product  Teams  notified  monthly  about  their 
laboratory  utilization  performance 
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SIL  Utilization  Planning  Perl 

OCTL  Integration  La 


01/07/2005 


iy 

“  <9^  x'l^^  x> 

<S^  <s^  <s^  # 

Week  Ending  (Date) 

Positive  =  Worked  more  Shifts  than  Planned 
Negative  =  Worked  less  Shifts  than  Planned 
Worksheet:  Worksheet  1;  11/28/2005 


Remedial  Actions 
Implemented 
07/08/2005 


£ 

£ 

W 

"Jo 

3 

tJ 

< 

> 

■o 


SIL  Utilization  Planning  Performance 

OCTL  Integration  Lab 


.S 

.S 


#  o^'  jy  jy  jy  jy 

Week  Bid'ng  (Date) 


Positive  =  Worked  more  Shifts  than  Planned 
Negative  =  Worked  less  Shifts  than  Planned 


A  50%  reduction  in 
unplanned  shifts 
A  18%  reduction  in 
variability 
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PDF  Complete. 
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If'ages  and  Expand 


Features 


Cards 


■  Flight  Test  Card  Deck  Preparation 

■  Time  consuming  process 

■  Incomplete  data  provided  from  test  plan 

■  Too  much  pulling  of  info  required  to  build  deck 

■  Last  minute  changes  disrupt  process 

■  Development  efforts  force  last  minute  input 

■  Process  not  well  defined  or  documented 

■  Customer  perception  of  incomplete  planning  efforts 

■  Customer  request  for  more  time  to  review  flight  cards 
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Card  Development 


Process  Title 


Flightiest  Card  Development 


Process  Definition 


Gather  flight  test  requirements,  write  the  Test  Point  (Test  Card )  steps,  plan  and  write 
the  mission  profile,  assemble  the  deck  and  receive  review  approval 


Input 


•VCRM 

•  Integrated  Test  Plan 

•  Flight  (Detailed)  Test  Plan 

•  Test  Cards  * 


Sub-Processes  Steps 


AObtain  objectives  and  requirements 
AOevelop  test  card  from  approved  inputs 
APrepare  for  and  conduct  reviews 
ACirculate  Flight  Deck  for  signature 
AConduct  Technical  Brief  and  distribute 
test  cards 


Output 


AAccurate  flight  deck  (mission 
profile  and  test  cards) 
ASufficient  Joint  Test  Force 
(JTF)  review  of  flight  deck 
prior  to  flight 


Applicable  Procedures 


ATechnical  Mission  Support  i  Flight  Card  Preparation 


Applicable  Tools 


•  Microsoft  Word,  Archived  Test  Cards,  reviews  and  meetings 

*  *  Test  cards  are  not  always  provided  by  the  project  and  are  written  by  the  test  conductor  MOHTHHOR  GHUMMAiNf 
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nt  Focus 


Completed  brainstorming  session  for  process 
improvements 

■  Immediate  implementation  of  priority  items 

Process  highlights 

■  Documented  process  with  roles  and  responsibilities 

■  Defined  input  requirements 

■  Required  test  card  review  prior  to  submitting  deck  for 
approval 

Early  deployment  of  new  Sector  test  card 
development  procedure 

■  Start  date  advanced  from  October  to  June 
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■  Lead  Time  for 
Customer  Review 


Reduce  Rediines 
at  Tech  Brief 


Test  Steps  Redlined  at  Tech  Brief  (%  of  total) 

565-03  574-03 


January-  Oct 2006 


A  43%  reduction  in  errors 
A  47%  reduction  in  variability 
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jineering 


Generation,  review  &  release  of  engineering  drawings 
is  the  fundamental  business  process  in  Vehicle 
Engineering 

The  release  process  is  key  to  ensuring  drawing  quality 
&  minimizing  future  rework 

■  Like  peer  reviews  in  the  system/software  world 

2006-2007  initiative  featured  improvements  to  the 
release  of  Direct  Drawing  Changes 

■  Follow-on  to  2005  initiative  to  improve  the  release  of 
new  drawings 

■  Initiatives  cover  electro-mechanical  &  vehicle 
subsystem  drawings 
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New  Drawings  (ND) 


Process  Titl 


Process  Definitio 


Review  and  Release 


This  process  focuses  on  the  review  and  release  of 
New  Drawings  (ND)  in  Vehicle  Engineering 


A^rawing 

Release 

Requirements 


Review  ND 
Release  ND 


A\pproved  ND 
A^hecklists 
^Configuration 
Control 


Direct  Drawing 
Changes  (DDC) 


Applicable  Procedu 


AnGC  Procedures,  Contractual  documents 


Process  li 


Process  Definitio 


Review  and  Release 


This  process  focuses  on  the  review  and  release  of  Direct 
Drawing  changes  (DDCa)  in  Vehicle  Engineering. 


S93HSE 


A)esign-  CATIA,  HarnesSys,  CAD  AM,  BIDS 
^KM-  lEDB,  Metaphase,  TcE 


Note:  A  similar  process  is  used  for 
release  of  Engineering  Orders  (EOs). 

Due  to  the  wider  variability  among  EO 
types/groups,  EO  baselines  are  still 
under  development. 
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AnGC  Procedures,  Contractual  documents 


m 


A)esign  -  CATIA,  HarnesSys,  CADAM,  BIDS 
^KM  -  lEDB,  Metaphase,  TcE 
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nprovement  Focus 


Created  and  Utilized 
DDC  Checklists 

Leveraged  improved 
engineering 
database  for  new 
DDC  data  collection 


Crew  &  Equipment  Checklist 

Oaviing  Number  HC99A 
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Candidates  selected  based  on  Pareto  analysis 

■  Processing  of  discrepancy  inspection  reports  (DIRs)  for 
nonconforming  items 

■  Review  of  engineering  drawings 

■  Response  to  field  service  engineering  requests 
(FSERs)  from  field  service  reps 

■  Response  to  seller  issues 

Process  improvement  opportunities  noted  & 
implemented  for  DIR  processing  and  FSER  response 

First  3  baselines  utilize  extensions  of  the 

author^s  log-cost  model 
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Category  (Cat)  Codes: 

A  -  Test  Required.  Avionics  Performs 
O  -  Test  Required.  Other  Dept  Performs 
N  -  No  Test  Required 
C  -  Configuration  Issue.  DIR  Remains 
Open 

P  -  “Park”.  DIR  Remains  Open  Awaiting 
Supplier  Repair/Parts,  Lab/Aircraft 
Time,  Management  Decision,  Etc. 
Typically  Used  For  an  Interim 
Disposition  and  May  Occur  at  Any 
Point  of  the  Process. 
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Revised  existing  Avionics  work  instruction 

Optimized  Manager/Tech  Lead  DIR  notification  and 
assignment;  instituted  assignment  cross-check  to 
ensure  same  day  assignment 

Implemented  weekly  status  reporting  &  review  by 
Avionics  management 

■  Automated  management  follow-up  for  DIRs  open  for  5 
days 

■  Implemented  Category  fPofor  DIRs  in  work  by  other 
groups  (Vendor,  Lab  Ops,  etc.) 

Conducted  training 
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Board 
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routine  Air  Force  Technical  Orders  (AFTO  Type  22)  into 
the  Joint  Integrated  Maintenance  Information  System 
(JIMIS)  was  a  reiaxed  schedule 

In  2005,  Northrop  Grumman  transitioned  to  a  Totai 
System  Support  Responsibility  (TSSR)  sustainment 
contract 

■  On-time  delivery  became  a  component  of  the  TSSR  award  fee 

■  The  AFTO  22  delivery  requirement  was  reduced  by  57%  with  the 
new  spec  limit 
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AFTO  Disposition  and  Incorporation  Process 


Process 

Overview 


Process  Definition 


_  Air  Force  Technical  Order  (AFTO)  Form  22  is  the  method  by  which 

the  government  recommends  changes/  improvements  to  Technical  Manuals.  Northrop 
Grumman  dispositions  and  incorporates  the  AFTOs  Issued  bv  the  government  Into  Manuals. 


Input 


AFTO  22  submitted  by 
JTF 

AFTO  22  Submitted  by 
116‘h  Wing 


Sub-Processes  Steps 


•  NG  at  Warner  Robins  dispositions  AFTO 
•LKS  Review  &  Approval  of  AFTO 

•  Processing  Days  In  LKS 

•  Develop  Data  Changes  In  LSA  Melbourne 

•  Incorporate  AFTO  into  JIMIS 

•  Review  Time  in  Pubs  Tech  Support 
•Gov’t  Review  in  Live  Feed 
•Release  of  Data 

•Data  Fielded  for  use 


Output 


Tech  Orders  fielded  for 
usage  by  the  116*^  wing 


Applicable  Procedures 


AFTO  Disposition  and  Incorporation  Procedure 


Applicable  Tools 


JIMIS  Database,  AFTO  Database  (Access) ,  Management  tracking  tool  (Excel) 
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In  2004,  analysis  was  conducted  on  that  year’s  entire 
data  set 

■  Of  all  data  points  at  or  above  the  new  spec  limit: 

■  67%  resulted  from  Improvement  AFTOs 

■  33%  resulted  from  Correction  AFTOs 

Although  not  conclusive,  preliminary  anaiysis 
suggested  that  the  two  subgroups  might  have 
different  distributions 

■  This  would  indicate  they  should  be  charted  separately 

Process  improvements  focused  on  improving  the 
assignment  &  management  of  open  AFTO  items 
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■  40%  reduction  in  the  LSA  throughput  time 

■  24%  reduction  in  the  process  variability 
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JIMIS  is  a  complex,  interactive  relational  database 

■  Integrated  electronic  technical  manual  (lETM) 

Database  Size  ~  7.5  GB 

■  >  1 00,000  pgs  of  text 

■  Replaces  -  400  technical  manuals 

Used  to  maintain  Joint  STARS  aircraft 

■  116*'^  Wing  at  Warner  Robins 

JIMIS  data  development  -  DCMA  rated  high  risk 
process 

■  Manned  aircraft 

■  Database  changes  affect  muitipie  aircraft 

■  Errors  in  maintaining  data  can  have  serious  consequences  on 
weapon  system  performance 

Government  reviews  new/changed  data  for  quality 

■  ^  400  submitted  in  each  reiease  cycie  (every  75  days) 

Contract  imposes  quality  performance  targets 
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Improved  review  process 

■  Expanded  scope  of  review 

■  Increased  standardization  of  review  methods 

■  Instituted  face-to-face  review  feedback  meetings 

■  Synchronized  timing  of  Government  review  with 
completion  of  internal  review 

Better  match  of  reviewers  expertise  to  components 
reviewed 

Automated  tracking  of  review  status 
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SPC  techniques  are  broadly  applicable  in  any 
engineering  disciplines 

Controiiing  &  improving  key  business  metrics  yield 
measurable  benefits 

■  Cost 

■  Schedule 

■  Quality 

Simple  process  improvements  work  in  the  reai  worid 

■  Standardization 

■  Oversight 

■  Automation 

■  Training 
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Northrop  Grumman  Corporation 

(321)951-5072 

Rick.Welch(g)ngc.com 
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5  Major  Sites, 
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5,221  Engineers, 

1  Data  Repository: 
Having  data  you  can 
ictually  use  -  Priceless! 
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Customer  Success  Is  Our  Mission  is  a  trademark  of  Raytheon  Company. 
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m  Introduction  to  Raytheon 

■  Measurement-related  Goals 

■  Measurement  Process  Overview 

■  Best  Practices 

T  Measurement  Definition 
T  Measurement  Collection 
T  Measurement  Analysis 
i  Tooling/Automation 

■  Future  Opportunities 

■  Results 

■  Q  &  A 
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Raytheon  and  NCS 


Raytheon 


■  Raytheon  is  an  industry  leader  in  defense  and  government 
electronics,  space,  information  technology,  and  technical 
services 


■  Network  Centric  Systems  (NCS)  develops  and  produces 
mission  solutions  for  networking,  command  and  control, 
battle  space  awareness,  homeland  security  and  air  traffic 
management  _ 
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Raytheon 


ANCS  Engineering  Organization  =  Over  5,000  individuals 
A  Number  of  programs  to  appraise  =  33  (CA  8,  TX  4,  IN  9  ,  FL  4,  MA  8) 
A  Various  leveis  of  CMMI  maturity  at  the  project  onset 
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|(elated  Goals 


Raytheon 


Establishing  a  Common  Measurement  Program 

T  All  major  NCS  sites  and  engineering  disciplines 
T  Common  plans  and  work  instructions  that  support  CMMI  Level  5 

V  Common  process  and  tooling 

Consistent  Approach 

T  Define  core  set  of  engineering  measures 

V  Define  analysis  that  should  occur  at  various  levels 

V  Define  measures  roll-up  as  related  to  NCS  goals 

V  Define  a  set  of  CMMI  Level  4  Sub-process  approaches 


■  Have  a  fbne  companyolook  to  our  customers 
i  Accurate  historical  data  and  consistent  estimates  across  sites 
i  Support  Mission  System  Integrator  (MSI)  role 
i  Support  multi-site  bids  and  work  transfers  between  sites 
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■  Cost  and  Schedule  Measures 

■  Defect  Containment 

■  Staffing  Profile 

■  Measurement  Compliance  - 

■  Change  Management 

■  Peer  Review 

■  Requirements  Volatility 

■  Design  Margin  Index  (DMI) 

■  Size 

■  Productivity 


Raytheon 


There  were  many  more, 
measures,  but 
Engineering  started 
with  a  iist  of  core 
measures 
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ACTIVITY  TITLE 

PE 

SE 

SW 

HW 

General 

Flardware 

Analog 

Digital 

FPGA 

Mechanical 

PROJECT  PLANNING  &  MANAGEMENT 

Planning  and  Management 

Quality  Engineering 

Configuration  Management 

REQUIREMENTS  DEVELOPMENT 

System  Requirements  Definition 

System  Design  &  Architecture 

Product  Requirements  Definition 

Product  Design  &  Architecture 

Component  Requirements  Definition 

PRODUCT  DESIGN  &  DEVELOPMENT 

Requirements  Management 

Simulation  and  Modeling 

Preliminary  Design 

Detailed  Design 

Implementation 

Integration 

SYSTEM  INTEGRATION  &  VALIDATION 

Product  Verification  &  Validation 

System  Integration 

System  Acceptance  Test 

System  Field  Test 

Aligns  disciplines  and 
activities 

Used  to  identify  and 
collect  costs  for  Work 
Breakdown  Structure 
(WBS)  elements 

Scheme  is  aligned  with 
Cost  Estimation 

Facilitates  collection  of 
consistent  historical  data 

Defect  data  can  be 
collected  in  these  bins 


Sets  the  foundation  for  CMMI  Level  5  by  aligning  cost,  schedule,  and  quality  data 
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es  have  Consistent  Elements 


■  Size  measures  were  defined  for  Systems  Engineering  (SE),  Software  (SW), 
Hardware  (HW)-Electrical,  HW-FPGA  (Field-Programmable  Gate  Array),  and 
HW-Mechanical  disciplines 

R  Sizes  for  each  discipline  were  defined  to  have  the  capability  to  be  converted  to 
equivalent  size  units,  where  equivalent  means  equivalent  to  requiring  the  same 
amount  of  effort  as  developing  it  from  scratch 

R  Each  discipline®  size  data  includes  these  elements 
I  Reused 
T  Modified 
I  New 

T  Reuse  Factor  (Fr) 

T  Modified  Factor  (Fm) 


Equivalent  =  New  +  (Modified  *  Fm)  +  (Reused  *  Fr) 
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■  Raytheon  created  the  SECOST  toe  I,  which  aids  deploy  men  ;  and  company 
calibration  with  the  Constructive  Systems  Engineer  ng  (Dost  Model  (COSYSMO) 

■  NCS  System  Engineering  sizes  are  aligned  with  CCiSYSMC'  sizes 


For  each  system  of  interest  these  are  collected  o  comi: 


requirements  (EREQ): 

■  System  requirements 

■  System  interfaces 

■  System  algorithms 

■  System  scenarios 


ute  (Equivalent 


■  For  a  complete  SE  size  set  of  requirements  data,  additional  NCS  SE  size 
measures  include: 


■  Software  product  requirements 

■  Hardware  product  requirements 

■  Hardware  component  requirements 
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SE  Full  Life  Cycle  Productivity 


Raytheon 


Business 

Strategy 


Planning  & 
Management 


Requirement 
&  Architecture 
Development 


Design  & 
Development 


Integration, 
Verification 
&  Validation 


Production 


& 

Apport 


SE  Specific  Life  Cycle  Stage  Productivities 


A_^ 

(  ^ 

I  1 

1 

(  ^ 

System 

Requirements 

Definition 

System 
Design  & 
Architecture 

Product  ^ 
Requirements 
Definition 

Product 
Design  & 
Architecture 

Component 

Requirements 

Definition 

Specific  cost  collection  codes  are  used  to  capture  hours 
_ for  Productivity  measures _ 
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■  Raytheon  has  used  parametric  SW  models  such 
as  COCOMO,  COCOMO  II,  RE  VIC,  Price-S,  and 
SEER-SEM  for  many  years 

■  Specific  alignment  was  made  to  the  SEER-SEM 
SW  Application  types  to  allow  stratification  of  data 
such  as  productivity 

■  NCS  SW  Size  measures  support  these  models 
with  parameters  of  Source  Lines  of  Code  (SLOC)  ' 
categorized  by  Reused,  Modified,  and  New,  with 
Reuse  and  Modified  Factors 

■  A  standard  NCS  software  line  counting  tool  was 
deployed  across  all  sites  so  that  sizes  are 
measured  consistently  and  with  automation 
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SW  Development 
Productivity  Stages 


Rqmts  & 

Prelim. 

Detailed 

Implement¬ 

Integration 

Verification 

Production 

§ps.  & 

Arch.  Devel. 

Design 

Design 

ation 

&  Validi^llpn 

%PPirt 

1 . . 

1  :  :li 

Specific  cost  collection  codes  are  used  to  capture  hours 
_ for  Productivity  measures _ 
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HW  Sub- 
Discipline 

Size  Unit 

Definition  of  Size  Unit 

Electrical 

Terminations 

Termination  count  is  the  sum  of  ail 
external  physical  leads 

FPGA 

FPGA  Lines  of 
Code 

Lines  of  Code  -  like  software 
engineering 

Mechanical 

Square  F-e(?t  of 
Drawing 

The  square  feet  of  drawings  required  to 
document  the  design 

Hardware  Size  Units  are  an  indication  of  which  hardware 
sub-discipline  is  producing  this  data 
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Raytheon 


HW  Development 
Productivity  Stages 


Reqs  & 
Architecture 
Devei. 

Prelim. 

Design 

Detailed 

Design 

Implement¬ 

ation 

Integration 

Verification 
&  Validation 

Production 

Ops.  & 
Support 

1 . a . 

ill  -S 

. . 

Collected  separately  for: 
AElectrical 
AfPGA,  and 
AMechanical 
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Quantitative  Anaiysis 
Program  Activities 


MTBF  V  Mean  Time 

Between  Failures 
AUPC  I  Average  Unit 

Production  Cost 
DP  MO  V  Defective  Parts 
per  Million 

_ Opportunities 


Standard  Process,  Tools, 
Enablers,  Technology 


Inspection  Calculator: 
Peer  Review  Defect  Density 
Review  /  Development  Stage 


Price-H: 

AUPC 

Design  for  Cost 


ASENT/BlockSIM: 

MTBF 

Requirements  Analysis/ 
Design  for  Reliability 


Legend 

AtooI 

Avieasure 

ySSub- 

process 


Cost,  DPMO 
Design  for  Cost  / 
Design  for  Producibility 


SECOST: 

Effort  hours  by  stage 
Development  Stage 


Programs  have  a  variety  of  toois  and  modeis  to  use  for  statisticai  controi 
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Raytheon 

Peer  Review  Example 


Program 
Execute  Peer 
Reviews 


Programs 

select 

baselines,  use 
control  charts, 
and  analyze 
capability 
monthly 


Programs 
record  Peer 
Review  data 
in  review  tools 


Program  Baseline 


Measurement 

Repository 


Establish  Org 
Baselines 


Org  Baseline 


■  Programs  use  latest  org  baselines  and  program/product  line  baselines 

■  Baselines  are  recalculated  periodically  and  then  fed  back  to  programs 

■  Peer  review  tools  are  updated  to  include  new  org  norms 
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Raytheon 


KPP  -  Key  Performance 
Parameters  are  system 
level  attributes 


TPM  -  Technical 
Performance  Measures 
are  functions  of  Key 
Product  Characteristics 


KPC  I  Key  Product 
Characteristics  can 
significantly  affect  a 
TPM  or  KPP 


■  KPPs  are  decomposed  into  objectives  and  managed  at  lower  levels  to  ensure  program 
success 

■  DMI  is  an  index  used  to  measure  the  design  margin 

■  DMI  is  a  useful  measure  for  assessing  “over”  design  and  “under”  design _ 
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>ver  Program  Life  Cycle 


Raytheon 


Requirements  Decomposition 

r  ^ 


Requirements  Definition  Design 

^  \  r 


Mfg/T est 


Form,  Fit,  or 
Functional 
Limit(s) 


Customer 

Requirement 


System 


Product 
(i.e.  Line  repiac- 
able  units. 
Subsystems, 
Etc.) 

Component 
(i.e.  Circuit 
Cards, 

Cabies,  etc.) 


KPCc 

KPCc 

KPCc 

I 


Aliocate 
‘Cpk  Target” 
(Cpk^) 


USL 


Cpk^ 


LSL 


Cpk^ 


LSL 


USL 


Cpk^ 


USL 


Cpk^ 


Predict 
“Current  Cpk” 
(Cpkp) 


TV 

Lp 

K 

^p  M 

T  y 

p 

.  USL 

\ 

USL 

Measure 
“Actuai  Cpk” 
(Cpk|^) 


\  JL  USL  \  ^  USL 


USL  LSL 


\ 


I^M 


USL 


■TPMs  are  used  for  quantitative  management  and  statistical  control 

■This  gives  the  programs  added  value  and  can  help  significantly  reduce  program  costs 
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Analysis  &  Review:  Raytheon 

ative  Management  Stakeholders 


Program  Engineer  and 
Discipline  Teams 


Site  Measurement  Teams 


NCS  Engineering  Process 
Steering  Team 


NCS  Measurement  Council 


Engineering  Councils 


■High  level  teams  and  managers  were  very  interested  is  analyzing  and  reviewing 
measurement  data 

■This  created  a  positive  “pull”  for  information  across  NCS 
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Analysis  &  Review: 
and  Review  Flow 


Raytheon 


Prog  Engr  Leads 
Perform  Analysis 


PE  Analysis 


SE  Analysis 


SW  Analysis 


HW  Analysis 


Management  Reports 


Review  with 
Program 
Mgmt 


Management  Reports 


Site 

Rolls-up 
&  Analyzes 
Data 


Analysis 

comments, 

Baselines, 

Predictive 

Models 


Trends.  Baselines. 


&  Analysis  Results 


Review 
with  Site 
Engr  Mgmt 


Analysis 

comments. 

Baselines, 

Predictive 

Models 


Generates  reports 
for  reviews 


Org 

Measurement 

Repository 


Analysis 
comments. 
Baselines, 
Predictive 
Models 


Org 

Rolls-up 
and  Analyzes 
Data 


Organizational 

Feedback 


Trends, 

Baselines, 

& 

Analysis 

Results 


Review  with 
NCS  Mgmt 
&  Engr 
Councils 


Consistent  flow  across  NCS  sites  and  disciplines 
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>mate  Databases  and  Tools 


Raytheon 


Automation  allows  repeatable  quick  entry  of  data  tools  to  supply  measurement  data! 
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jnities 


Raytheon 


■  Increase  the  coverage  and  use  of  common  cost  collection 
codes  to  more  disciplines  and  activities 

■  Extend  use  of  measurement  database  to  other  roll-up 
management  measures  such  as  Oregon  Productivity 
Matrixes  (OPMs) 

■  Incorporate  statistical  and  textual  analysis  capability  into  the 
measurement  reporting  automation 

■  Improve  alignment  of  financial  processes  and  tooling  with  the 
common  cost  collection  codes 


■  Define  collection  scheme  for  the  Incremental  Development 
life  cycle  model 

■  Continue  to  broaden  the  scope  of  automation  that  supports 
collection  and  reporting  or  measures 
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Cooliemon,  LLC  ^ 

compfeted  a 

Capability  Maturity  Model  Integration  (CMMI*)  Appraisal 
On  6/1/2007 

According  Lo  the  Software  Engineering  Institute  (SEI) 
Slanciard  Appraisal  VItthod  for  Process  Improvemenl  vl.l 

SCAMPD^^  -  Class  A 

and  rated 


^R^iytheon  ^Networ^Centric  Systems 

(s^/sw/m^ 


iCMMP^  Maturity  Level  5 


Ralph  Willianm 

SEI  Authorized  Lead  An?rniscr 
a)  0UK10073-(.l() 


Raytheon  NCS  deploys 
integrated  processes  with 
measures  across  multiple 
disciplines  and  sites  to  an 
engineering  org  of  over 
5,000  !!! 


CMMI 
Level  5 


Raytheon  NCS  Achieves  CMMI  Level  5  on  1  June  2007  for 
Systems,  Software,  and  Hardware  Engineering  ! 
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Chris  Angermeier 

(NCS  TX  Measurement  Lead) 

I  972.952.3679 

)'  c-anaermeier@ravtheon.com 

Jill  Brooks 

(NCS  TX  SW  Process  Technical  Director) 
I  972.344.3022 
I  iill  a  brooks@ravtheon.com 


Raytheon 
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Applied  to  Software  Requirements 
Specification  Process 

Al  Florence 
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♦  Introduction 

»  Background  of  Statistical  Process  Control 

♦  Overview  of  Software  Engineering  Institute 

»  Capability  Maturity  Model  Integration 
»  Quantitative  Project  Management  and  related  Process  Areas 

♦  Statistic  Process  Control 

»  Overview  of  Control  Charts 

♦  Examples  of  Control  Charts 

»  Applied  to  the  Requirements  Specification  Process 

♦  Conclusion 

♦  Contact  Information 

♦  Acronyms/Abbreviations 
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♦  Statistical  Process  Control  (SPC)  has  been  applied  to  manufacturing 
processes  very  effectively  for  many  years. 

♦  Recently  software  organizations,  with  higher  process  maturity  levels,  have 
started  to  apply  SPC  to  their  software  development  processes. 

♦  Applying  SPC  to  requirements  efforts  sets  the  stage  for  applying  it  to 
subsequent  development  activities. 

♦  This  may  provided  the  biggest  pay-off  since  most  problems  in  software 
engineering  can  be  directly  traced  to  improper  definition  and  specification  of 
requirements. 
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♦  Capability  Maturity 

♦  CMMI®  Level  4  -  Quantitative  Project  Management 

»  SG  2  Statistically  Manage  Sub-process  Performance 

>  The  performance  of  selected  sub-processes  within  a  project’s  defined  process 
is  statistically  managed. 

-  SP  2.1  Select  Measures  and  Analytic  Techniques 

-  SP  2.2  Apply  Statistical  Methods  to  Understand  Variation 

-  SP  2.3  Monitor  Performance  of  the  Selected  Sub-processes 

-  SP  2.4  Record  Statistical  Management  Data 


CMMI  is  a  registered  trademark  of  the  SEI 
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♦  CMMI®Other  Process  Areas 

♦  CMMI®  Level  5  -  Causal  Analysis  and  Resolution 

»  SG  1  Determine  Causes  of  Defects 


>  Root  causes  of  defects  and  other  problems  are  systematically  determined. 

-  SP  1 .1  Select  Defect  Data  for  Analysis 

-  SP  1 .2  Analyze  Causes 

»  SG  2  Address  Causes  of  Defects 

>  Root  causes  of  defects  and  other  problems  are  systematically  addressed  to 
prevent  their  future  occurrence. 

-  SP  2.1  Implement  the  Action  Proposals 

-  SP  2.2  Evaluate  the  Effect  of  Change 

-  SP  2.3  Record  Data 
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♦  The  intent  of  SPC: 

»  Is  to  better  understand  and  monitor  process  behavior  and  to  bring  it  under 
control  when  required. 

»  Is  not  necessarily  to  monitor  products  per  se,  although  this  maybe  a  by¬ 
product  of  SPC. 
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Determine  Cause  of  Deviation 


Upper  Control  Limit  -  ULC 


Measurements 


3  Standard  Deviations  (+3  sigma) 


Center  Line  -  CL 


3  Standard  Deviations  (-3  sigma) 

_ 

Determine  Cause  of  Deviation 
Time  - 


Lower  Control  Limit  -  LCL 


♦  According  to  the  normal  distribution,  99%  of  all  normal  random  values  lie 
within  +/-3  standard  deviations  from  the  norm 

♦  If  a  process  is  under  Statistical  Process  Control,  all  measurements  should 
fall  within  the  3-sigma  limits 

♦  If  not,  the  anomaly  needs  to  be  investigated  for  cause  and  the  process 
brought  back  under  control 
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♦  Control  charts: 

»  Separate  signal  from  noise 

>  so  when  anomalies  occur  they  can  be  recognized 

»  Identify  undesirable  trends 

>  they  point  out: 

-  Fixable  problems 

-  Potential  process  improvements 

»  Show  the  capability  of  the  process 

>  so  achievable  goals  can  be  set 

»  Provide  evidence  of  process  stability 

>  which  justifies  predicting  process  performance 
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♦  Control  charts  use  two  types  of  data: 

»  variables  data 
»  attributes  data 

♦  Variables  data  are  usually  measurements  of  continuous  phenomena  such 
as: 

»  elapsed  time 
»  effort  expended 
»  memory/CPU  utilization 
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♦  Attributes  data  are  usually  measurements  of  discrete  phenomena  such  as: 

»  number  of  defects 
»  number  of  source  statements 
»  number  of  people 

♦  Most  measurements  in  software  used  for  SPC  are  attributes  data. 
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♦  The  following  are  control  charts  that  should  be  used  for  variables  data  and 
for  attributes  data: 

»  Attributes  Data 

>  u  charts 

>  Z  charts 

>  XmR  charts 


»  Variables  Data 

>  X-bar  charts 

>  R  charts 

>  XmR  charts 


Al  Florence 

11 


MITRE 


♦  u  charts  are  used  when  the  data  are  samples  from: 

»  a  Poisson  distribution,  and 
»  the  areas  of  opportunity  are  not  constant 

♦  Z  charts  can  be  used  to  avoid  variable  control  limits  for  both  large  and  small 
variations 
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♦  XmR  charts  can  be  useful 

»  when  little  is  known  about  the  underlying  distribution,  or 

»  when  the  justification  for  assuming  a  binomial  or  Poisson  process  is 
questionable 

♦  X-bar  and  R  charts  are  used  to  portray  process  behavior  when  you  have  the 
option  of  collecting  multiple  measurements  within  a  short  period  of  time 
under  basically  the  same  conditions 
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♦  Other  sigma  limits  for  homogeneous  sets  of  data  (The  Empirical  Rule) 
»  1  sigma 

>  Roughly  60%  to  70%  of  data  will  be  located  within  1  sigma 

»  2  sigma 

>  Roughly  90%  to  98%  of  data  will  be  located  within  2  sigma 

»  3  sigma 

>  Roughly  99%  to  1 00%  of  data  will  be  located  within  3  sigma 
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♦  Tests  for  out-of  control  situations 
»  Test  1 


>  A  single  point  falls  outside  the  3-sigma  control  limits 

»  Test  2 

>  At  least  2  out  of  3  successive  points  fall  on  the  same  side  of,  and  more  that  2- 
sigma  units  from,  the  center  line 

»  Tests 

>  At  least  4  out  of  5  successive  points  fall  on  the  same  side  of,  and  more  that  1- 
sigma  unit  from,  the  center  line 

»  Test  4 

>  At  least  8  successive  values  fall  on  the  same  side  of  the  center  line 
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♦  A  government  agency,  while  re-developing  legacy  systems,  reverse 
engineered  the  existing  software  requirements 

♦  Five  teams  were  assigned  to  reverse  engineer  related  sets  of  functional 
requirements 

♦  This  author  was  assigned  as  a  consultant  to  support  the  agency  in  the 
proper  specification  of  the  requiremente 
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♦  The  examples  illustrate: 

»  the  proper  specification  of  requirements 

>  Specification  in  this  context  means  “writing”  the  requirements 

»  the  application  of  control  charts  applied  to  the  requirements  specification 
process 
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cipation 


What  is  wrong  with  this  requirement? 

After  the  system  receives  the  Validation  file,  the  system  shall: 

A  notify  the  individual  about  acceptance  or  rejection. 

A  the  acceptance  file  must  contain  the  name  and  ZIP  code  of  the 
individual. 

A  rejected  validation  request  must  include  the  Reason  Code. 
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ecifying  a  Good  Requirement  (i  of  4) 

5  The  following  are  some  critical  attributes  that  requirements  must  adhere  to: 

(used  to  critique  the  requirements) 

♦  Completeness:  Requirements  should  be  complete 

»  They  should  reflect  system  objectives  and  specify  the  relationship  between 
the  software  and  the  rest  of  the  subsystems 

♦  Consistency:  Requirements  must  be  consistent  with  each  other;  no 
requirement  should  conflict  with  any  other  requirement 

»  Requirements  should  be  checked  by  examining  all  requiremenb  in  relation 
to  each  other  for  consistency  and  compatibility 
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^  Feasibility:  Each  requirement  must  be  feasible  to  implement 

»  Requirements  that  have  questionable  feasibility  should  be  analyzed  during 
requirements  analysis  to  prove  their  feasibility 


♦  Traceability:  Each  requirement  must  be  traceable  to  some  higher-level 
source,  such  as  a  system-  level  requirement 

»  Each  requirement  should  also  be  traced  to  lower  level  design  and  test 
abstractions  such  as  high-level  and  detailed-level  design  and  test  cases 
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ecifying  a  Good  Requirement  (3  of  4) 

5  Testability:  All  requirements  must  be  testable  in  order  to  demonstrate  that 
the  software  end  product  satisfies  its  requirements 

»  In  order  for  requirements  to  be  testable  they  must  be  specific, 
unambiguous,  and  quantitative  whenever  possible.  Avoid  negative,  vague 
and  general  statements 

♦  Unique  identification:  Uniquely  identifying  each  requirement  is  essential  if 
requirements  are  to  be  traceable  and  testable 

»  Uniqueness  also  helps  in  stating  requirements  in  a  clear  and  consistent 
fashion 
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ecifying  a  Good  Requirement  (4  of  4) 

5  Design  Free:  Software  requirements  should  be  specified  at  a  requirements 
level  not  at  a  design  level 

»  The  approach  should  be  to  describe  the  software  requirement  tunctionally 
from  a  system  (external)  point  of  view,  not  from  a  software  design  point-of- 
view,  i.e.  describe  the  system  functions  that  the  software  must  satisfy. 

♦  Use  of  “shall”  and  related  words:  In  specifications,  the  use  of  the  word  "shall" 
indicates  a  binding  provision 

»  Binding  provisions  must  be  implemented  by  users  of  specifications.  To 
state  non-binding  provisions,  use  "should"  or  "may".  Use  "will"  to  express  a 
declaration  of  purpose  (e.g.,  "The  Government  will  furnish..."),  or  to  express 
future  tense.  MIL-STD-490A 
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♦  It  needs  to  be  noted  that  requirements  do  not  “live  alone” 

»  They  depend  on  other  requirements  and/or 
»  on  clarifying  comments 

to  present  a  complete  view  of  the  functionality  associated  with  a  related  set 
of  requirements. 

♦  A  related  set  of  functional  requirements  may  be  introduced  with  a  preamble 
describing  the  capability  of  the  functional  set. 

»  The  preamble  does  not  itself  establish  requirements;  this  is  done  later  in  the 
requirements’  specifications. 

♦  Some  requirements  may  be  amplified  with  clarifying  comments  which  are, 
again,  not  part  of  the  requirements,  but  add  understandability. 
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♦  Some  requirements  are  documented  sequentially  with  the  requirements  stated 
first  setting  the  “stage”  for  the  following  requirements  which  add  more  and  more 
capability. 

»  The  later  stated  requirements  depend  on  the  earlier  requirements  to  complete  their 
functionally. 

»  An  example  may  be  the  use  of  the  word  “processing”.  If  the  processing  of  a 
functional  set  of  related  requirements  has  been  described  in  earlier  requirements 
the  later  requirements  may  amplify  and/or  reference  the  processing  without  having 
to  restate  the  processing. 

♦  This  is  the  case  in  the  following  examples;  they  have  been  extracted  from  a 
larger  set  of  functionally  related  requirements  and  may  not  present  a  complete 
picture  of  the  entire  set. 

♦  If  a  single  requirement  was  to  be  a  complete  picture  of  a  complex  capability, 
one  requirement  would  have  to  describe  the  entire  capability  making  it 
extremely  complex  and  difficult  to  understand,  implement,  and  test. 
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♦  The  first  set  of  requirements  were  received  from  the  teams  before  they  had 
been  exposed  to  the  critical  attributes  while  the  subsequent  sets  were  received 
after  they  had  incorporated  review  comments  and  had  been  trained  on  using 
the  attributes. 


♦  Later  sets  of  requirements  still  had  defects  which  were  detected  in  subsequent 
critiques  and  used  to  create  the  control  charts  related  to  those  iterative  sets. 

»  This  continued  for  several  months  until  it  was  felt  that  the  process  was  under 
statistical  process  control  and  that  requirements  were  well  specified. 

»  Because  of  this  some  readers  may  want  to  find  additional  issues  associated  with 
these  examples,  other  than  the  ones  listed  in  the  critiques. 

»  Also,  there  may  be  issues  with  the  re-specifications,  but  keep  in  mind  that  these 
hopefully  would  be  identified  in  subsequent  critiques. 
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♦  The  following  examples  illustrate  the  application  of  SPC  to  the  process  of 
specifying  requirements 

♦  The  first  two  examples  show  some  requirements 

»  As  initially  specified  by  the  teams 

»  Followed  by  this  authors  critique  against  the  critical  attributes  of 
requirements 

»  The  re-specification  of  the  requirements 


Each  violation  against  the  critical  attributes  will  be  recorded  as  a  defect  to 

be  used  to  construct  control  charts. 
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♦  The  next  three  examples  show  control  charts  applied  to  the  specification  of 
the  requirements 

»  The  first  control  chart  example  depicts  the  requirements  specification 
process  as  being  out  of  statistical  process  control 

»  The  next  control  chart  shows  the  process  on  the  path  towards  being 
brought  under  control 

»  The  third  one  shows  the  process  under  statistical  process  control 
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txampie  i  (i  of  2) 


Initial  specification: 

3.4.6.3  The  system  shall  prevent  processing  of  duplicate  electronic  files 
by  checking  the  SDATE  record.  An  e-mail  message  shall  be  sent. 

Critique: 

1 .  Two  “shalls”  under  one  requirement  number. 

2.  When  is  the  SDATE  record  checked? 

3.  Against  what  other  records  is  the  SDATE  record  checked? 


MITRE 


4.  What  is  checked  in  the  SDATE  record? 


8  critical  attributes 
violations  (defects) 


5.  To  whom  is  the  email  message  sent? 

6.  What  does  the  email  message  say? 

7.  When  is  the  email  message  sent? 

8.  The  requirement  has  design  implications,  SDATE  record. 

>  A  requirement  should  specify  what  the  data  in  the  record  are  and  not  the 
name  of  the  record  as  it  exists  in  the  design  and  implementation 
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♦  Re-specification: 

3.4.6.3  The  system  shall: 

a.  Prevent  processing  of  duplicate  electronic  files  by  immediately  checking 
the  date  and  time  of  the  submission  against  prior  submissions,  and 

b.  Immediately  send  the  following  e-mail  message  to  submitter: 

1 .  Request  updated  submission  date  and  time,  if  necessary,  and 

2.  State  that  the  submission  was  successful,  when  successful. 
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♦  Initial  specification: 

♦  After  the  system  receives  the  Validation  file,  the  system  shall: 

»  Notify  the  individual  about  acceptance  or  rejection 
»  The  acceptance  file  must  contain  the  name  and  ZIP  code  of  the  individual 
»  Rejected  validation  request  must  include  the  Reason  Code 

♦  Critique: 

1 .  The  second  and  third  bullets  don’t  make  sense,  try  to  read  them  as  such: 

>  the  system  shall  the  acceptance  file  must... 

>  the  system  shall  rejected  validation... 

2.  Use  of  both  “shall”  and  “must” 

3.  Where  are  the  reason  codes? 

4.  Who  is  notified? 

5.  How  is  the  individual  notified? 

6.  No  unique  identifier 

MITRE  7.  Use  of  bullets,  bullets  are  difficult  to  trace  Al  Florence 


7  critical  attributes 
violations  (defects) 


3.27.3  When  the  system  receives  a  validation  file,  the  system  shall: 

a.  Reject  the  file  if  it  does  not  contain  the  individual’s 

1.  name,  and 

2.  ZIP  code,  and 

b.  Notify  the  individual  via  electronic  transmission  about  acceptance  or  rejection 
with  a  reason  code  for  rejection.  (Reference  Reason  Code,  Table  5.4.8),  and 

c.  Request  corrected  resubmission,  if  rejected. 


MITRE 


Al  Florence 

31 


^  Complete 


r 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


txampie  ^  (i  of 


4) 


Out  of  Statistical  Process  Control 

♦  Example  3  will  show  a  control  chart  of  all  teams'  attempts  at  the  initially 
specification  of  the  requirements 

♦  This  was  before  they  received  guidance  on  the  criticai  attributes 


Raw  data  collected  from  the  initial  specification  of  the  requirements 


Teams 


Totals 


MITRE 


No.  Rqmts 


105 


134 


98 


201 


196 


734 


Defects 


305 


172 


105 


205 


407 


1194 


*DefectsX100/ 
No  of  Rqmts 


290.48 


128.36 


107.15 


101.16 


207.66 


Vefects  normalized 
to  100  requirements 
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Calculations  to  be  used  to  construct  the  control  chart 


♦  Plot  =  Number  of  defects  X 1 00  /  requirements  specified  [calculated  for  each 
team’s  data] 

♦  CL  =  (total  number  of  defects/total  number  of  requirements)  X 1 00 

♦  UCL  =  CL+3(SQRT(CL/a1)  [calculated  for  each  team’s  data] 

♦  LCL  =  CL-3(SQRT(CL/a1)  [calculated  for  each  team’s  data] 

♦  a1=  Requirements  specified/100  [calculated  for  each  team’s  data] 
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txampie  ^  (3  of  4) 


Calculations  to  be  used  to  construct  the  control  chart 


Teams 


Plot 


CL 


UCL 


LCL 


1 


290.48 


162.67 


200.01 


125.33 


1.05 


128.36 


162.67 


195.72 


129.62 


1.34 


107.15 


162.67 


201.32 


124.03 


0.98 


101.10 


162.67 


189.66 


135.68 


2.01 


207.66 


162.67 


190.00 


135.34 


1.96 
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mpie  ^  (4oT4) 


Control  Chart  for  the  Initial  Specification  of  Requirements 


Defects 


♦  For  control  charts  to  be  valid,  they  need  to  be  used  on  processes  that  are  mature  and 
conducted  consistently  and  on  measurements  that  are  valid,  i.e.  correctly  depict  the  process 

♦  This  control  chart  showed  that  the  process  was  immature  and  out  of  statistical  process 
control 

♦  The  teams  had  not  received  guidance  on  the  critical  attributes  of  requirements,  i.e.,  were  not 
following  a  consistent  process 
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Toward  Being  Brought  Under  Statistical  Process  Control 


♦  Example  4  will  show  a  control  chart  of  all  teams'  subsequent  attempts  at  the 
specification  of  the  requirements.  New  sets  of  requirements  were  inciuded. 

♦  The  teams  had  been  trained  in  the  criticai  attributes  and  most  had  resoived 
the  critique  issues 
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Raw  Data 


Teams 


1 

2 

3 

4 

5 

Totals 


No.  Rqmts 


Defects 


98 

125 

107 

198 

205 

733 


35 

139 

45 

85 

95 

399 


DefectsXI  00/ 
No.  of  Rqmts 

35.71 
1 1 1 .20 
42.06 
42.93 
46.34 


Calculations 
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3) 


Control  Chart  for  Subsequent  Specification  of  Requirements 


An  anomaly  occurred  with  the  second  team’s  effort 


Causal  analysis  revealed  that  the  second  team  had  not  implemented  the 
critique’s  findings  nor  analyzed  new  requirements  against  the  critical  attributes. 
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Under  Statistical  Process  Control 


♦  Example  5  will  show  a  control  chart  of  all  teams'  subsequent  attempts  at  the 
specification  of  the  requirements.  New  sets  of  requirements  were  inciuded. 

♦  Management  ensured  that  the  second  team  resoived  the  issues  identified  in 
the  critique  and  that  they  anaiyze  additionai  requirements  against  the  criticai 
attributes. 
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Raw  Data 


Teams 


Totals 


No.  Rqmts 


105 


116 


101 


205 


298 


825 


Defects 


14 


35 


DefectsXlOO/ 
No.  of  Rqmts 


1.90 


3.45 


5.94 


4.39 


4.70 


Calculations 


When  the  LCL  is 
negative 
it  is  set  to  zero. 
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Control  Chart  for  Subsequent  Specification  of  Requirements 


The  requirements  specification  process  is,  for  now,  under  statisticai  process  controi. 
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conclusion 


♦  The  examples  demonstrate  the  use  of  SPC  applied  to  the  requirements 
specification  process.  Many  more  control  charts  were  constructed  and 
analyzed.  The  ones  use  here  were  selected  to  succinctly  demonstrate  their 
use. 

♦  The  use  of  statistics  using  SPC  control  charts  and  other  statistical  methods 
can  easily  and  effectively  be  used  in  a  software  setting.  SPC  can  identify 
undesirable  trends  and  can  point  out  fixable  problems  and  potential  process 
improvements  and  technology  enhancements. 

♦  Using  SPC,  beginning  with  requirements  analysis,  can  provide  the  biggest 
payoff.  It  is  a  well-known  fact  that  if  requirements  are  properly  defined  early 
in  the  development  life  cycle,  the  migration  of  problems  into  the  later  phases 
will  be  mitigated. 
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♦  CL  -  Center  Line 

♦  CMMI®  -  Capability  Maturity  Model  Integration 

♦  ET  -  Eastern  Time 

♦  FA  -  Financial  Agent 

♦  LCL  -  Lower  Control  Limit 

♦  SPC  -  Statistical  Process  Control 

♦  UCL  -  Upper  Control  Limit 


MITRE 


Al  Florence 

46 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


Rayllieon 

Customer  Success  Is  Our  Mission 


Using  the  Scientific 
Method  to  Achieve 
Levei  4  and  5 

Inferential  Statistical 
Models  and  their 
Relationship  to  CMMI 
Levels  4  and  5 

Jeff  N.  Ricketts,  Ph.D. 


Copyright  ©  2007  Raytheon  Company.  All  rights  reserved. 
Customer  Success  Is  Our  Mission  is  a  trademark  of  Raytheon  Company. 


^PDF 

't  Comolete 


use  period  has  ended. 
Thank  you  for  using 


Your  complimentary 


PDF  Complete. 


od  and  inferential 
ed 


Raytheon 


irat/e  to 

and  Expanded  Features 


The  scientific  method  is  a  body  of  techniques  for  investigating 
phenomena,  acquiring  new  knowledge,  or  correcting  and 
integrating  previous  knowledge.  It  is  based  on  gathering 
observable,  empirical  and  measurable  evidence  subject  to 
specific  principles  of  reasoning.  The  scientific  method 
consists  of  the  collection  of  data  through  observation  and 
experimentation,  and  the  formulation  and  testing  of 
hypotheses.  The  scientific  method  is  used  to  explain  and 
predict  the  causes  of  variability  in  natural  phenomena. 

Inferential  statistics  or  statistical  induction  comprises  the  use 
of  statistics  to  make  inferences  concerning  relationships 
within  a  population.  These  relationships  are  expressed  in 
causal  terms 
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The  standard  measures  commonly  in  use  today  all  have  one 
thing  in  common:  they  are  historical  vs.  predictive 

They  are  all  reactive  vs.  proactive 

Some  metrics  have  little  relationship  with  the  real  questions 
that  need  to  be  answered 

Corrective  actions  are  usually  haphazard  and  unverifiable  as  to 
their  effectiveness 

There  are  no  standard  measurement  definitions 
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We  need  to  do  a  better  job  applying  scientific 
methods  and  inferential  statistical  models  to  our 
business  to  determine  what  causal  relationships 
exist  between  the  variables  that  we  can  control  in 
order  to  optimize  our  processes  and  tools  and 
reduce  development  costs 

Level  4-5  processes  can  be  optimized  through  the 
use  of  causal  analysis  and  predictive  measurement 
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QPM  SG2-  Statistically  Manage  Sub  process  Performance 
SP2.1  -  Select  Measures  and  Analytic  Techniques 
SP2.2  -  Apply  Statistical  Methods  to  Understand  Variation 
SP2.3  -  Monitor  Performance  of  the  Selected  Sub  processes 
SP2.4  -  Record  Statistical  Management  Data 

OPP  SP1 .5  -  Establish  and  maintain  process  performance  models  for  the  organization®  set  of 
standard  processes 

OID  SG1  -  Select  Improvements 
SP1.3  -  Pilot  Improvements 

CAR  SG1  -  Determine  Causes  of  Defects 
SP1 .1  -  Select  Defect  data  for  Analysis 
SP1 .2  -  Analyze  Causes 

CAR  SG2  -  Address  Causes  of  Defects 
SP2.1  -  Implement  Action  Proposals 
SP2.2  -  Evaluate  the  Effect  of  Changes 
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The  product  development  process  consists  of  many  variables  (tools, 
people,  processes,  inputs,  outputs) 

There  is  a  lot  of  variation  in  these  factors  and  consequences  of  the 
variation: 

stability  of  requirements 
makeup  of  peer  review  teams 
stability  of  design 

types  of  tools  and  technology  used 
number  of  defects  identified  in  peer  reviews 
amount  of  hrs  of  training  per  engineer 
maturity  of  technology 
types  of  development  environments  used 
skill  sets/mix 

programming  language  or  design  methods  used 
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X  seems  to  happen  more  often  when  Y  is  around 

We  always  seem  to  do  better  when  we  use  this 
product/method/tool/process 

Do  we  really  save  time  by  conducting  formal  peer  reviews  for 
reused  and  ported  code? 

Are  peer  reviews  even  necessary  on  a  product  line? 

Use  cases  take  a  long  time  to  develop.  Are  they  really 
necessary? 

The  key  is  to  identify  factors  that  appear  to  be  associated  with 
each  other  or  are  not  reducing  cost  and  schedule 
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If  you  suspect  that  there  is  a  causal  relationship 
between  two  variables,  the  relationship  is  stated  in 
the  form  of  hno  differenceo 

e.g.  Systems  engineers  find  the  same  number  of 
defects  during  peer  reviews  as  software  engineers. 

e.g.  The  amount  of  preparation  time  one  takes  for  a 
peer  review  has  no  relationship  to  the  number  of 
defects  identified 
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Measurements  must  be  consistent,  precise  and  repeatable 

Measures  are  targeted  for  the  type  of  statistics  that  will  be 
generated 

Nominal  -  categorical/dichotomous-  systems  engineers  vs. 
software  engineers 

Ordinal  -  categorical  -low  medium  high-  complexity  factors, 
lift/mod/reuse 

Interval  -  frequency  distributions-  ten-  years  of  experience 
Ratio  -  frequency  distributions  with  an  absolute  zero 
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ategory  of  data 


Nominal 

Difference  in  proportions, 

Chi  square,  Lambda, 
student®  t  test 

Ordinal 

Analysis  of  Variance, 
Exactness  tests.  Rank 

Order  correlation,  Gamma 

Interval 

Correlation  and  regression. 

Multiple  and  stepwise 
regression,  path  analysis 

Ratio 

Correlation  and  regression, 
multiple  and  stepwise 
regression,  path  analysis 
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Samples  must  be  representative  of  the 
population  under  study 

Samples  must  be  randomly  selected  (  can  be 
simple,  stratified,  cluster,  etc) 

Samples  cannot  be  the  whole  population 

Statistics  computed  must  be  appropriate 
for  the  level  of  measurement 
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What  is  the  observed  difference  between  Group  A 
and  Group  B? 

What  is  the  measure  of  association  between  the 
independent  variable  (X)  and  the  dependent 
variable  (Y)? 

Significance  levels  tell  you  if  the  observed 
difference  is  statistically  significant 

Given  no  relationship  between  what  you 
measured,  this  is  the  probability  (.05,  .01,  .001) 
that  you  would  observe  this  result  in  a  randomly 
drawn  sample  from  the  population  of  this  size? 
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Study  1 


Study  2 


Xj-Xj  =0 


Xi—X2  =0 


x^=1425  X2=2L75 


^  =3Q0  3^  =3475 


N2=  25 


N2=  450 


Group  1  was  composed  only  of  requirements  developers 
Group  2  was  composed  of  testers  and  requirements 
developers 


Which  observed  difference  between  these  groups  is 
statistically  significant  given  their  sample  sizes? 
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What  is  a  line  of  code? 


What  is  a  defect? 


What  is  productivity? 


What  is  rework? 


What  is  a  requirement? 
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Changes  in  X  appear  to  be  causing  changes  in  Y 
when  in  fact  Z  is  associated  with  both  X  and  Y  so 
when  Z  varies  both  X  and  Y  vary 
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♦  We  could  be  doing  a  much  better  job  and  adding  more 
value  to  our  level  4-5  processes  by  incorporating  the  use  of 
the  scientific  method  and  inferential  statistical  models  into 
our  measurement  and  analysis  processes 

♦  The  data  is  there,  but  being  collected  inconsistently 

♦  Random  samples  allow  us  to  create  probability  distributions, 
generate  sample  statistics  and  to  test  null  hypotheses  that 
will  aid  us  in  being  able  to  predict  the  effect  of  fine  tuning 
our  methods  used  to  build  our  products  and  Dispel  myths 
and  non  truths  regarding  the  value  of  non-value  added 
tasks. 

♦  Statistically  significant  results  typically  warrant  further 
investigation 

♦  Correlation  is  not  necessarily  causation 
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Process  Performance  Models 


SOLUTIONS  TOGETHER 


A  FEW  SIMPLE  STEPS 


1 .  Determine  what  you  are  trying  to  accomplish! 

2.  Identify  the  activities  involved  in  accomplishing  the 
objective. 

3.  Understand  how  much  the  activities  impact  the 
outcome. 

4.  Gain  a  statistical  understanding  of 
performance  of  key  activities. 

5.  Do  the  math. 

6.  Model  the  objective. 

7.  Use  the  model. 

8.  Rinse  and  repeat. 
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SOLUTIONS  TOGETHER 


Definitions  (my  version) 


A  Objective  T  something  you  are  trying  to  accomplish 


The  things  being  done  are 
meaningless  until  put  in 

the  context  of  the  objective. 
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Process  Performance  Models 


SOLUTIONS  TOGETHER 


STEP  NUMBER  1 


A  Determine  what  you  are  trying  to  accomplish! 
I  What  is  the  objective? 
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STEP  NUMBER  1 


Company  XYZ 


Increase  Sales  in  Customer  Service  area  by 
selling  more  features  to  existing  customers. 

Why  arencbthey  already  doing  this? 

NO  TIME!!! 

Refined  Objective:  Create  more  time  for  customer  service  reps  to  have 
available  for  selling  features  to  existing  customers. 
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Definitions  (my  version) 


A  Sub- Process  or  process  elements  i  the  activities 
involved  in  obtaining  an  objectivi 


The  things  being  done  are 
meaningless  until  put  in 
the  context  of  the  objective. 
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SOLUTIONS  TOGETHER 


STEP  NUMBER  2 


A  Identify  the  activities  involved  in  accomplishing 
the  objective. 

I  This  could  be  an  iterative  step  depending  on 


the  objective 
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REQUIREMENTS 

DEVELOPMENT 

-Q 


ELICITING 


to 


-Q 


DRAFTING 


to 


-Q 


REVIEWING 


to 


-Q 
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CUSTOMER  MEETING 
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SPEC  TRANSLATION 
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INTERNAL 

SPEC  REVIEW 
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.'X 

CUSTOMER 

REVIEW 

APPROVING 
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T ricks  to  Step  2 


Break  the  activities  down  to 
something  that  can  be  controlled 

V  Attendance 

V  Amount  of  material 
T  Amount  of  time 

I  Etc. 
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STEP  NUMBER  2  Example 


CUSTOMER 

SUPPORT 


-Q 


RESEARCH 


to 


-Q 


HELP  DESK 


to 


-Q 


TRAINING 


-Q 


to 


-Q 


LOCATE  INFORMATION 


to 


COLLATE  MATERIALS 


to 


REPORT  RESULTS 


to 


OTHER 


OTHER 
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STEP  NUMBERS 


A  Understand  how  much  the  activities  impact  the  outcome 


I  Many  statistical  techniques  available  to  ascertain  this,  if 
necessary 


Company  XYZ 

How  much  time  is  being  spent  in  each  of  these  Research  Activities? 


Pareto  Chart  of  Research  Activities 
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Research  Activities 


Count 
Percent 
Cum  % 
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71.4 
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CT 
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May  want  to  use  sampling 
techniques  for  initial  data 
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STEP  NUMBER  4 


A  Gain  a  statistical  understanding  of  the  historical 
performance  of  key  activities 

I  Typically  use  Control  Charts  for  this,  or  some  type  of  historical 


Company  XYZ  historical  results 
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STEP  NUMBERS 


A  Do  the  Math! 


I  Locate  Information  =  52.4%  of  Research  Time 
I  Total  Research  Time  =  65%  of  Customer  Support  Time 
I  Need  to  Increase  available  time  by  1 5% 

I  Total  CS  Hours  currently  are  5500 


Cut  fLocate  Informationo 
time  by  535  hours 
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STEP  NUMBER  6 


A  Model  the  Objective! 

T  May  need  to  include  multiple  activities  and  process  areas  to  put 
together  the  best  picture  for  meeting  the  objective. 

V  At  this  point  we  are  really  trying  to  understand  how  changes  to 
the  process  activities  impact  the  objective  or  target 


Company  XYZ 


How  do  we  define  Sales  Availability  as  a  function  of  fLocate  Informationc? 
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SOLUTIONS  TOGETHER 


STEP  NUMBER  6  Example 


If  it  takes  on  average: 

A  29  hours  to  locate  info 
A  30  hours  to  locate  info 
A  25  hours  to  locate  info 
A  20  hours  to  locate  info 


Work  Mgmt 


Customer  Support 
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USING  THE  MODELS 


A 


A 

A 


Understand  quantitatively  \Nhat  needs  to  change,  if 
anything,  in  order  to  reach  the  objective 

i'  How  much,  exactly,  do  we  need  to  change?  (from  29  to  20 
hrs  to  flocate  informationoV  sets  the  specification) 

i'  Maintain  a  statistical  understanding  of  the  current 
performance  of  key  activities 

\  The  best  way  to  ensure  you  will  not  exceed  spec  is  to 
monitor  average  and  variation  in  control  chart 

Monitor  the  execution  of  the  process  activities  in  order  to 
ensure  consistent  execution 

Regularly  input  process  activity  values  into  model 
equation  to  ascertain  current  status  to  objective 
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A 
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RINSE  AND  REPEAT! 


SOLUTIONS  TOGETHER  ^ 


A  Be  aware 

T  No  model  will  be  naccurateothe  first  time  through,  but 
it  will  still  provide  information 

T  A  few  iterations  must  occur  before  you  will  adequately 
understand  relationships  between  process  activities 
and  objectives 

T  Continue  monitoring  process  activities  in  order  to 
ensure  consistency  of  execution 

T  The  more  unstable  your  process  execution,  the  less 
predictable  your  model  will  be 


't  Comolete 
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Thank  you  for  using 
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SOLUTIONS  TOGETHER  7 - ui  nthct 


A  Post  release  defects  as  a  function  of  amount  of 
material  inspected 

A  Schedule  impacts  as  a  function  of  customer 
attendance  at  requirements  reviews 

A  Cycle  time  as  a  function  of  reused  components 

A  Rework  budget  as  a  function  of  design  inspection 
prep  time 


A  YOUR  MODEL  WILL  VARY!!! 
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Questions  or  Comments? 
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SOLUTIONS  TOGETHER 


For  More  Information 


A  Technical  questions: 

I  Virginia  Slavin,  703-742-7131 , 
i'  slavin@systemsandsoftware.org 
A  For  services,  training  requests,  account  information: 
I  Hillary  Davidson,  703-742-7188 
V  davidson@systemsandsoftware.org 
A  For  Consortium  products  or  general  questions: 

I  Contact  Clearinghouse  (ask-spc@software.org) 

T  or  800-827-4772 

A  If  you  are  a  Consortium  member,  go  to 

www.software.org  and  select  For  Members  Only  to 
download  documents  or  view  product  websites  (will 
automatically  get  you  on  newsletter  mailing  list) 
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■  Raytheon  is  an  industry  leader  in  defense  and  government 
electronics,  space,  information  technology,  and  technical 
services 


■  Network  Centric  Systems  (NCS)  develops  and  produces 
mission  solutions  for  networking,  command  and  control, 
battle  space  awareness,  homeland  security  and  air  traffic 
management  _ 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


tes 


Raytheon 


ANCS  Engineering  Organization  =  Over  5,000  individuals 
A  Appraised  as  CMMi  Levei  5  for  Systems,  Hardware  and  Software 


Engineering  is  June,  2007 
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■  Per  Webster.com,  productivity  is: 


Raytheon 


Main  Entry: 

T  pro-duc-tiv-i-ty 

Pronunciation: 

T  \Dpr6-dek-Dti-ve-te,  Qpra-,  pre-Ddek-\ 

Function: 

T  noun 

Date: 

V  circa  1810 

1  :  the  quality  or  state  of  being  productive 

2  :  the  rate  per  unit  area  or  per  unit  volume  at  which  biomass 
consumable  as  food  by  other  organisms  is  made  by 

producers 


26  July  2007  Page  5 


^PDF 

Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


Productivity  (continued) 


Raytheon 


In  the  manufacturing  world, 
productivity  is  number  of 
widgets  created  per  time 

Use  productivity  as  input  for 
estimation  and  planning:  If  we 
know  we  can  produce  X 
widgets  /  hour,  and  we  have 
an  order  for  1 0OX  widgets, 
then  it  will  take  us  100  hours 
to  meet  the  order 
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Raytheon 


■  Also  use  productivity  to  aide 
with  analysis  regarding 
program  progress,  if  CPI  (Cost 
Performance  Index)  and  SPI 
(Schedule  Performance  Index) 
appear  to  be  good,  the 
program  could  still  have  issues 
if  productivity  is  not  near  what 
was  originally  planned.  Rolling 
up  measurements  can  mask 
issues 


Analyze  metrics 
holistically! 
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Raytheon 


■  Increased  productivity  can  be 
used  as  a  measure  of  process 
improvement,  if  all  else  is  held 
constant 

■  Let®  look  at  an  examplee  . 
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■  Productivity  is  measured  as  size  per  time 
such  as  meters  /  second 


If  you  change  the  size,  the  time  will  have  to 
change,  assuming  that  productivity  remains 
constant  (and  it  is 
fairly  constant  at 
the  Olympic  level) 


In  the  Olympic  sprint  events,  the  distance  is  the  reizeothat  is 
producedS  so  the  200  meter  dash  is  twice  as  far  as  the  1 00 
meter  race 


Photos:  Credit  Getty  Images 


26  July  2007  Page  9 


jzzle: 
oductivity 

Productivity  =  Size  /  Hours 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


Raytheon 


Size  =  ELOC  =  Equivalent  Lines 

Hours  =  SW  development  hours 
=  (ACWPctd  +  ETC) 


ACWPqtd  =  Actual  Cost  of  Work  Performed  (cumulative  to  date) 


ETC  =  Estimate  to  Complete 

=  the  remaining  hours  expected  to  complete  the  work 
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Size  data  includes  these  counts,  in  lines  of  code,  or  thousands 
of  lines  of  code,  KSLOC 

V  New:  Any  software  or  firmware  unit  that  is  to  be 
newly  developed  or  does  not  fit  the  reused  or  mod¬ 
ified  software  definitions 

V  Reused:  If  no  lines  of  the  actual  component  code  are  going  to  be 
changed.  This  includes  comments.  If  the  component  is  to  be  edited 
for  any  reason,  it  cannot  be  classified  as  reused.  If  the  component  is 
to  be  converted  to  a  different  language,  it  cannot  be  classified  as 
reused 

V  Modified:  Estimated  SLOG  modifications  for  that  component  do  not 
exceed  50%  of  the  actual  counted  SLOG.  If  the  SLOG  modifications 
exceed  50%  of  the  actual  size,  the  effort  associated  with 
understanding  and  modifying  the  component  is  likely  to  be  equal  to 
or  exceed  the  effort  required  to  develop  it  new,  so  treat  it  as  new 


26  July  2007  Page  1 1 


Raytheon 


^PDF 

Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


uzzle:  Size  (cont.) 


Size  data  includes  these  factors: 


V  Reuse  Factor  (Fr):  Fr  is  the  factor  for  converting  reused  code 
(SLOC  to  ELOC).  it  represents  the  percent  of  overaii  effort  that  the 
estimator  beiieves  wiii  be  required  to  adapt  the  existing  software 
component  and  artifacts,  versus  developing  the  software  and  ali 
associated  artifacts  from  scratch 


Modified  Factor  (Fm):  F^^  is  the  factor  for  converting  modified  code 
(SLOC  to  ELOC).  it  represents  the  percent  of  overaii  effort  that  the 
estimator  beiieves  wiii  be  required  to  adapt  the  existing  software 
component  and  artifacts,  versus  developing  the  software  and  ali 
associated  artifacts  from  scratch 


26  July  2007  Page  12 


Raytheon 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


uzzle:  Size  (cont.) 


■  Delivered  Lines  of  Code: 

DLOC  =  New  +  Reused  +  Modified 

■  Equivalent  Lines  of  Code: 

ELOC  =  New  +  (Modified  *Fm)  +  (Reused  *  Fr) 


10 

01 

100 

OlOlO 

lOOIOlO 

OlOIOOlO 

lOOIOOIOI 

OlOIOlOOlOO 

OOlOlOlOIOOlO 

lOOIOIOlOOlOOl 


ELOC  is  generally  used  for  productivity  as  it  results  in  a  more 
representative  measure 
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m  You  cand  attribute  an  increase  in  productivity  to  reuse 


■  Reuse/modification  means  that  there  is  less  work  to  do  or, 
going  back  to  the  Olympic  Sprint  analogy,  less  distance  to 
cover 


■  The  productivity  equation  takes  this  into  account  using  the 
Reuse  and  Modified  factors 
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■  Raytheon  has  used  parametric  SW  models  such 
as  COCOMO,  COCOMO  II,  RE  VIC,  Price-S,  and 
SEER-SEM  for  many  years 

■  Specific  alignment  was  made  to  the  SEER-SEM 
SW  Application  types  to  allow  stratification  of  data 
such  as  productivity 

■  NCS  SW  Size  measures  support  these  models 
with  parameters  of  Source  Lines  of  Code  (SLOC)  ' 
categorized  by  Reused,  Modified,  and  New,  with 
Reuse  and  Modified  Factors 

■  A  standard  NCS  software  line  counting  tool  was 
deployed  across  all  sites  so  that  sizes  are 
measured  consistently  and  with  automation 

■  Also  aligned  with  customer  expectations  T  they 
often  use  these  models 
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uzzle:  Hours 


SW  Development 
Productivity  Stages 


Rqmts  & 

Prelim. 

Detailed 

Implemen¬ 

Integration 

Verification 

Product! 

ion  Ops.  & 

Arch.  Devei. 

Design 

Design 

tation 

&  Valicistionii : 

Siinnnff 

■N  X  V 

ACWPcjd  =  Actual  Cost  of  Work  Performed  (cumulative  to  date)  =  sum 
of  all  hours  charged  against  SW  Development  Productivity  Stages 

ETC  =  Estimate  to  Complete  =  the  remaining  hours  expected  to 
complete  the  work 


Specific  cost  collection  codes  are  used  to  capture  hours 
_ for  Productivity  measures _ 
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ACTiViTY  TiTLE 

PE 

SE 

sw 

1  HW  1 

General 

Flardware 

Analog 

Digital 

FPGA 

Mechanical 

PROJECT  PLANNiNG  &  MANAGEMENT 

— 

— 

— 

— 

— 

Planning  and  Management 

1 

1 

These  elements 
contribute  to  the 
denominator  in 
the  productivity 
eouation 

Quality  Engineering 

Configuration  Management 

REQUIREMENTS  DEVELOPMENT 

System  Requirements  Definition 

1 

System  Design  &  Architecture 

Product  Requirements  Definition 

Product  Design  &  Architecture 

: — ^ 

Component  Requirements  Definition 

/ 

PRODUCT  DESIGN  &  DEVELOPMENT 

n  ^ 

Requirements  Management 

Simulation  and  Modeling 

Preliminary  Design 

i 

f 

\ 

Detailed  Design 

I 

Implementation 

1 

1 

Integration 

' 

V  , 

/ 

SYSTEM  INTEGRATION  &  VALIDATION 

Product  Verification  &  Validation 

System  Integration 

System  Acceptance  Test 

System  Field  Test 

Aligns  disciplines  and 
activities 

Used  to  identify  and 
collect  costs  for  Work 
Breakdown  Structure 
(WBS)  elements 

Scheme  is  aligned  with 
Cost  Estimation 

Facilitates  collection  of 
consistent  historical  data 

Defect  data  can  also  be 
collected  in  these  bins 


Clearly  define  the  denominator  (e.g.  hours) 
_ in  the  productivity  equation _ 


26  July  2007  Page  17 


■  How  to  modification  of  existing  code/reuse  of  code 


Equivalent  =  New  +  (Modified  *  Fm)  +  (Reused  *  Fr) 


■  50%  or  less  modification  threshold,  or  counted  as  new 

■  If  many  products  are  at  50%  while  other  products  are  at  10%, 
wondthis  skew  the  data? 

■  No  changes,  used  as  is,  or  counted  as  modified/new 

■  Cost  of  integration,  and  verification/validation  will  vary  from 
product  to  product 

■  If  you  adjust  the  factors  to  account  for  this,  how  do  you  hround 
tripothe  data  to  ensure  that  your  estimates  will  improve?  Too 
many  variables,  not  enough  equations?  We  cand  really 
measure  the  factors 
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■  How  to  measure  productivity  of  non-traditional/partial 
lifecycles,  such  as  modeling  and  simulation  /  demo  products 
or  maintenance  versus  mission  software 


SW  Development 
Productivity  Stages 


f 

Ar 

^qmts  & 
ch.  Devel. 

Prelim. 

Design 

Detailed 

Design 

Implemen¬ 

tation 

Integration 

Verification 
&  Validation 

Proi 

duction 

Ops 

sap 

9.  & 

pm 

t 

■  May  not  fully  execute  all  activities/stages 

■  Flag  modified  lifecycle,  via  properties,  to  allow  stratification  to 
avoid  comparing  nappies  and  oranges© 
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m  How  to  handle  inclusion  of  COTS 


■  When  using  COTS,  there  is  no  effort  to  create  the  code,  but 
extensive  effort  can  be  spent  on  integration 

■  If  the  COTS  code  size  is  folded  in  with  ftraditionalocode  size, 
the  productivity  will  be  skewed 


One  solution  is  to  put  this  data  into  a  separate  fbucketoso  that 
it  can  be  evaluated  independently  and  then  a  factoring 
determined  so  that  it  can  be  rolled  up 


r 


Alternatively,  COTS  can  be  counted  as  Reused 
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■  How  to  handle  inclusion  of  autogenerated  code 

■  When  using  autogenerated  code,  the  effort 
spent  on  creating  the  code  itself  is  negligible 


■  If  the  autogen  code  size  is  folded  in  with  flraditionalocode 
size,  the  productivity  will  be  skewed 


■  One  solution  is  to  count  the  code  as  Reused  with  a  low  factor 


■  Alternatively  put  this  data  into  a  separate  fbucketoso  that  it 
can  be  evaluated  independently  and  then  a  factoring 
determined  so  that  it  can  be  rolled  up 
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■  Variation  in  measurement  of  size 


■  Not  all  using  the  same  line  counting  tool 

■  Not  measuring  at  the  same  level  of 
granularity  with  regard  to 
new/mod/reused 

■  Language  impacts  size 

■  Line  counting  tool  evolutions  handling 
historical  data 


■  Standardization/refine  of  organization 
tools/process  on-going 
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Variation  in  measurement  of  hours 


Raytheon 


■  Unpaid  Overtime  issue 

■  Supplier/Contractor  labor  ->  $ 
instead  of  hours 

■  Challenging  issues  due  to 
financial  policies  /  requirements  / 
tooling 
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Use  of  productivity  during  development  vs.  at  program 
completions  projected  vs  actual 

Limited  value  during  program 


Actuals  used  for  planning  and  estimating 
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Several  factors  contribute  to  the  calculation  of 
productivity 


Although  the  calculation 
of  productivity  is  fairly 
simple,  ensuring 
collection  of  appropriate 
data  and  the  use  of  the 
measurement  is  complex 


Solving  the  puzzle  of 
productivity  is  a 
continuing  journey 
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Jill  Brooks 

(NCS  TX  SW  Process  Technical  Director) 
I  972.344.3022 
I  iill  a  brooks@ravtheon.com 

Chris  Angermeier 

(NCS  TX  Measurement  Lead) 

I  972.952.3679 

)'  c-anaermeier@ravtheon.com 


Raytheon 
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A  Overview 

A  Measurement,  Expense  or  Investment 
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A  Staffing  and  Schedule 
A  Understanding  Trade-offs 
A  Conclusion 
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Overview 


DILBERT 


DOGBERT  CONSULTS 


YOU  NEED  A  DASH¬ 
BOARD  APPLICATION 
TO  TRACK  YOUR 
KEY  rAETRICS. 


E 

8 

r 


iS 


■o 


THAT  UAY  YOU^L  HAVE 
rAORE  DATA  TO  IGNORE 
LJHEN  YOU  rAAKE  YOUR 
DECISIONS  BASED  ON 
COrAPANY  POLITICS. 


i 

fS 

3 


Ji 


O 


o 


UILL  THE 
DATA  BE 
ACCURATE? 


OKAY, 

LET'S 

PRETEND 

THAT 

rAATTERS. 


Does  this  sound  familiar? 
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ent:  Expense  or  Investment 


A  Software  measurement  (and  process 

improvement)  are  viewed  as  expenses;  Overhead 

T  Lean,  agile  organizations  want  to  reduce  overhead 
T  But,  how  do  organizations  become  “lean  &  agile”? 

A  Part  of  cost  of  doing  business 

T  3  -  5%  on  average 
T  Project  management  averages  14% 
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A  What  does  software  measurement  provide? 

1.  Knowledge  of  an  organization’s  capabilities 

2.  Identifies  patterns  and  trends  (Strengths  to  leverage  and 
weaknesses  to  correct) 

3.  Insight  into  projects  in  time  to  make  effective  mid-stream 
corrections 

4.  Ability  to  benchmark  against  competition  or  “the  industry”  in 
quality,  productivity,  and  time  to  market 

5.  Quantitative  basis  for  evaluating  project  and  organizational 
performance 

A  Improves  ability  to  meet  commitments,  avoid 

pitfalls,  and  evaluate  trade-offs 
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)f  the  Industry:  Project 


Estimation 


A  Software  estimates  are  not  project  plans 

A  Estimates  contain  uncertainty  about  two  key 
components: 

T  Scope  of  the  requirements  (project  size) 

T  Team  productivity 
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Cone  of  Uncertainty 


Project  Definition 


Time 


a  5k 


1 


VartabNity  in  the 
Estimate  of 
Proiect  Scope 
(effort  cost  features) 


I.2Sk 

10k 

OOx 


Not  enough  information  is  available  early  in  the  development 
lifecycle  to  make  accurate  estimates 

Precision  is  not  accuracy 


*  The  Intelligence  behind 

Successful  Software  Projects 


©Quantitative  Software  Management,  Inc.  #7 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 

rade  to 

and  Expanded  Features 


al  vs.  Estimated  Effort 


Effort  Growth 


Actual  vs.  Estimated  Effort 
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VS.  Estimated  Schedule 


Schedule  Growth 


□  Percent 


Actual  vs.  Estimated  Schedule 
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In  Summary 


A  Average  schedule  growth  is  8% 

A  Average  cost/effort  growth  is  16% 

A  Average  size  growth  is  15% 

A  So  how  can  we  use  this  information  to  create  more 
accurate  estimates? 


w  T-he  Intelligence  beiiind 

Successful  Software  Projects 


©Quantitative  Software  Management,  Inc.  #11 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 
- 3rr 


deling  Increased  Size 


A  Create  best  project  estimate  based  on  proposed 
size 

T  Use  historically  based  productivity 
T  Account  for  project  constraints  (staff,  effort,  schedule) 

A  Create  estimate  based  on  15%  size  growth 

T  Does  this  account  for  projected  schedule  &  effort  growth? 
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15%  Growth  (575  FP) 
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Life  Duration  (Months)  Risk  Profiie 

<1  5%  size  grow  th> 


— I - 1 - 1 - 1 - 1 - 1 - 1 - T 

10  20  30  40  50  60  70  80 

Assurance  Level  (%) 


15%  Growth  (575  FP) 


Life  Effort  (PM)  Risk  Profiie 

<1  5%  size  grow  th> 


Averages  close  to  numbers  predicted  for  effort  and  schedule 
growth  (10.2  duration  and  43  staff  months  of  effort) 
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What  is  "normal"  variability? 
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A  Case  Study 


A 

A 

A 

A 

A 


838  projects  that  had  data  reported  for  Analysis/Design  as  well 
as  Construction  and  Test  phases 

Average  Effort  applied  to  Analysis/Design  =  20% 

474  projects  in  the  sample  used  <=  20%  design  effort 
y  Average  Analysis/Design  Effort  =  11% 

364  projects  in  the  sample  used  >20%  design  effort 
I  Average  Analysis/Design  Effort  =  33% 

Size  profiles  of  samples  very  similar 


A 

A 

A 

A 
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Observations 


A 


A 


Projects  with  <20%  effort  in  Requirements  and 
Design 

T  Took  12®/©  longer  to  complete 
T  Averaged  5.6 ®/o  more  effort  (median  24.4®/©  greater) 

T  Had  an  average  staff  14.6%  higher 

But  these  projects  did  excel  at  one  thing: 

T  Found  63.7®^  more  defects  in  systems  test 
T  Had  127®^  more  defects  in  the  first  12  months  after  delivery 
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erstanding  Trade-offs 


Size  =  Effort^  xTime^  x  Productivity 

where  a  =  ^  and  b  =  j 


Additional  schedule  has  a  much  larger  impact  on  a  software 
project  than  increased  effort 
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Imposs 


Estimating  Conundrum 
Schedule  /  Effort  Tradeoff 


Uncertainty  about  Size  and  Productivity  creates 
uncertainty  about  the  Duration-Effort  curve 


r 
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Duration 


GSM 
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Estimating  Conundrum 


Sometimes  no  Solution  Works 


Impossible  Zone 


Duration 
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Estimating  Conundrum 


Reduce  Size  (Functionality) 


Impossible  Zone 


Duration 
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Conclusions 


A  Measurement  is  an  integral  part  of  management 

A  Information  required  to  make  precise  estimates  is 
unavailable  at  project  start-up 

T  Estimate  uncertainty  decreases  rapidly  with  more  information 

A  Project  estimates  understate  effort,  schedule,  & 
size 

T  Estimating  based  on  a  larger  size  or  at  a  higher  assurance  level 
can  account  for  this 

A  The  trade-off  between  schedule  &  cost/effort  is 
non-linear 
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Conclusions 


A  Effort  spent  in  Analysis  &  Design  pays  big 
dividends 

T  Reduces  overall  project  effort  (cost$$$$) 

T  Reduces  overall  project  schedule 
T  Improves  project  quality 


T-he  Intelligence  betund 
Successful  Software  Projects 


©Quantitative  Software  Management,  Inc.  #28 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Questions 

9 


Ttie  Intelligence  behind 
Siiccessfijl  Software  Projects 


©Quantitative  Software  Management,  Inc.  #29 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Optimizing  the 
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Communications  and  information 
networks 


Aviation  electronics 


Intelligence,  surveillance,  and  Space  and  ground  satellite  Operations  and  support  services 

reconnaissance  communications  systems 


We  innovate,  integrate,  and  manage  technology. 
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Introduction 
I  Background 
I  Goals  and  Objectives 
T  Terminology 
I  Approach 
Roadnnap 

T  Characteristics  of  Success 
T  Measurement  Analyst 
I  User  Viewpoints 
T  Automation  as  an  Enabler 
I  Leading  Indicators 
Results 

T  Information  Needs 
T  Measurement  Objectives 
T  Executive  Management  Viewpoint 
T  Indicator  Improvements 
T  Lessons  Learned 
Summary 
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A  Harris  CMMI®  Level  3  compliant  since  11/2005 

A  Measurements  used  regularly  for  program 
monitor  and  control 

rt  Need  for  improvement  still  recognized 

A  Measurement  process  relies  on  manual  input 

A  Perception  too  many  measures,  some  measures 
redundant 

A  Management  desires  increased  emphasis  on  fact 
based  decision  making 
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Goals 


/-t  Improve  measurement  and  analysis 
effectiveness 


T  Enhance  measurement  infrastructure  to  improve 
A  Efficiency  &  value 
A  Predictability 
A  Competitive  advantage 

I  Reduce  quantity  of  measures  to  effectively  manage 
programs  and  align  with  division  objectives 

T  Increase  number  of  leading  indicators 

Improve  measurement  foundation  for 
advancement  to  CMMI®  Level  4  or  5 
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Objectives 


hj\Rms 


A  Develop  simple,  consistent,  reliable  measurements 

A  Reuse  or  modify  existing  measurements 

A  Provide  rapid  access  to  fresh,  actionable  information 

A  Examine  quality  and  completeness  of  data 

A  Increase  consistency  with  industry  standards 

A  Increase  predictability  of  program  execution 

A  Facilitate  straight-forward  and  objective  analysis  of 
measures 

A  Enable  automated  collection  of  data  and  creation  of 
indicators 

A  Evaluate  adequacy  of  existing  data  to  support  high 
maturity  analysis 
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tinology 
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Jser 

Viewpoint 


Measurement 

Specification 


Repository 

Content 


Examples: 

Information 

Program  CPI 

Product 

Chart 

t 

Indicator 

CPI  with 

Thresholds 

t 

Derived 

CPI  = 

Measure 

BCWP/ACWP 

t 

Base 

BCWP, 

Measure 

ACWP 

Attribute 


Cost 
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A  utilize  an  independent  industry  measurement 
expert  to  validate  and  achieve  maximum  results 

A  Identify  classes  of  measurement  users 

A  Define  information  needs  of  users,  based  on 

I  User  role  and  responsibilities 
T  Business  and  improvement  objectives 

A  Specify  indicators 

T  Define  leading  and  concurrent  indicators 
I  Use  existing  measures  where  possible 

A  Conduct  reviews  with  stakeholders 
A  Update  command  media 
A  Deploy  incrementally 
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A  Characteristics  of  Success 
A  Measurement  Analyst 
/  User  Viewpoints 
A  Automation  as  an  Enabler 
A  Leading  Indicators 
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Measures  based  on  business  goals 
A  Comprehensive  measurement  planning 
A  Measurement  expertise 

T  Training  in  defining,  collecting  and  analyzing 
measures 

T  Mentoring  and  advice 

Appropriate  resources 

I  Robust  tool  support 
I  Measurement  analysts 

A  Management  support 
A  Broad  participation 
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urement  Analyst  Role 


i^ARRES 


A  Use  of  measurement  is  a  part  of  everyone®  job 
A  Additional  expertise  maximizes  effectiveness 

\  Recognize  significant  trends 

V  Communicate  with  data  providers  and  decision  makers 

V  Efficient  &  consistent  execution  of  measurement  process 

Areas  of  expertise 

\  Design/Pian  measures  and  process 

V  Training  and  mentoring 

T  Anaiysis  and  interpretation  to  support  decision  makers 

A  Often  a  part  time  job 

T  Program  ievei  support 
\  Organizational  ievei  support 
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r  Automation 


A-  More  Timely  Access  to 
Data  and  Analysis 

T  Makes  data  immediately 
avaiiable 

V  Facilitates  driil  down  to 
investigate  anomalies 

V  Makes  information  avaiiable 
in  time  to  affect  business 
and  project  outcomes 

T  Facilitates  gathering  and 
anaiyzing  data  for  lessons 
learned 

T  Make  data  widely 
accessible 


Improved  Data  Quality 

T  Ensures  more  complete 
data 

T  Reduces  transcription 
errors 

T  Removes  redundancy  and 
inconsistency  in  data 
reporting 

T  Easily  supports  users  with 
different  information  needs 

Reduces  effort  for 

producing 

measurement  reports 
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A  Definition 

I  Has  predictive  value,  provides  early  warning  of  trouble 
(in  time  to  affect  the  outcome) 

A  Types  of  leading  indicators 

I  Observed  trends  predict  future  results  of  that  indicator 

I  Changes  in  one  indicator  predicts  future  results  of 
another  indicator 

I  Constraints  that  limit  performance 

A  Obstacles  for  leading  indicators 

I  Cumulative  measures  and  percentages 
T  Inconsistent  measurement  definitions 
I  Delays  in  data  collection  and  analysis 
I  Subjective  criteria  and  reporting 
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A  Information  Needs 
A  Measurement  Objectives 
A  Executive  Management  Viewpoint 
A  Indicator  Improvements 
A  Lessons  Learned 
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A  Program  Team  Members 

T  Implement  processes 
effectively 

T  Produce  quality  products 
T  Complete  tasks  on-time 

Program  Team  Leaders 

I  Estimate  and  plan 
T  Monitor  and  control 

Customer 

I  Monitor  product  quality 
T  Monitor  performance  to  plan 

T  Verify  appropriate  capability 
delivered  to  field 


Functional  Management 

T  Develop  improvement  plans 
with  measurable  objectives 

T  Improve  functional  processes 
across  projects 

T  Develop  staff  within  functions 

T  Provide  historical  data  for 
estimating 

Executive  Management 

V  Provide  program  oversight 
(project  by  project) 

V  Ensure  overall 
process/organizational 
health  (across  projects) 

T  Achieve  organizational 
financial  performance 
(across  projects) 
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A  Provide  program  oversight  (program  by  program) 

T  Meet  customer  expectations  &  satisfy  the  customer 
I  Produce  a  high  quality  compliant  product 
T  Perform  in  accordance  with  the  agreed  to  cost  &  schedule 
T  Meet  program  objectives 

Ensure  overall  process/organizational  health 
(across  programs) 

T  Increase  productivity  in  all  functions  (increase  effectiveness) 

T  Reduce  program  rework  (early  &  effective  removal  of  defects 
across  the  product  life  cycle) 

T  Increase  predictability  of  program  performance 
T  Increase  accuracy  of  program  estimates 
T  Maintain  CMMI  Level  3  maturity  rating 

T  Foster  a  rewarding  &  satisfying  work  experience  for  Harris 
employees 

Achieve  organizational  financial  performance 
(across  programs) 

T  Meet  Annual  Operating  Plan  (AOP)  objectives 
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^  Provide  program  oversight  (project  by  project) 

I  Meet  customer  expectations  and  satisfy  the  customer. 
Alechnical  Performance  Measures 
Ab  Risk  Summary 
A  Award  Fee  Graphs 
A  Customer  Satisfaction  Data 
I  Produce  a  high  quality  compliant  product. 

Ab  Defects  by  Phase 

A  Defects  Currently  Open  and  Total  Closed 
A  Defect  Severity  Tracking 
ATechnical  Performance  Measures 
Ab  Process  Compliance  Data 

b  indicates  leading  indicator 
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A  Provide  program  oversight  (project  by  project) 

T  Perform  in  accordance  with  the  agreed  to  cost  and 
schedule. 

Ab  Milestone  Progress 
Ab  Staffing  Tracking 
Ab  Requirements  Tracking 
AevMS  Tracking 

T  Deliver  the  expected  Return  on  Sales  (ROS)  on  the 
project. 

A  Investment  Profile 
A  Financial  Objectives 
ASales,  Order,  Profit  Tracking 
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i-y  Ensure  overall  process/organizational  health 
(across  programs) 

T  Increase  productivity  in  all  functions 

A  Efficiency  Measures 

T  Reduce  project  rework 

A  Rework  Effort  Tracking 

A  Defect  Phase  Containment  Tracking 

I  Increase  predictability  of  project  performance 

A  Earned  Value  Management  System  (EVMS)  Reports 

I  Increase  accuracy  of  project  estimates 

A  Project  Characterization  Worksheet  Analysis  by 
Function 
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H,  Ensure  overall  process/organizational  health 
(across  programs) 

T  Maintain  CMMi®  Levei  3  maturity  rating 
Ab  Process  Compiiance  Data 

T  Foster  a  rewarding  and  satisfying  work  experience  for 
Harris  empioyees 

AOrganizationai  Training  Reports 
AEmpioyee  Engagement  Surveys 
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A  Achieve  organizational  financial  performance 
(across  programs) 

T  Meet  AOP  objectives 
A  Investment  Profile 
A  Financial  Objectives 
A  Award  Fee  Tracking 
ASales,  Order,  Profit  Tracking 
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hy  Number  of  overall  Indicators  needed  was 
reduced 

A  Number  of  leading  indicators  was  increased 

Some  objective  indicators  added  to  balance 
subjective  indicators 
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i-y  Using  a  systematic  framework  helps  organize  the 
process 

A  Measurement  process  needs  to  evolve  with  the 
organization 

A  Tool  considerations  can^be  ignored 

A  Objective,  external  advice  helps  validate 

A  Expect  resistance  to  change 

A  Efficiency  measures  should  be  determined  by 
the  functional  organizations 
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A  CMMI®  compliance  doesn^  ensure  and  efficient 
and  effective  measurement  program 

A  A  systematic  approach  is  essential  to  balancing 
user  measurement  needs 

A  Next  Steps 

T  Develop  Executive  Management  viewpoint  first 
A  Set  expectations  for  leadership  &  program  teams 
A  Refine  business  objectives 
T  Develop  other  user  viewpoints  over  time 
T  Measurement  &  Analysis  training 

I  Develop  a  Business  Intelligence  (Bl)  architecture, 
design  and  deployment  plan 
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Harris  Corporation 
P.O.  Box  37 

Melbourne,  Florida  32902-0037 


http://www.harris.com/ 
SEI  Partner 


Gary  Natwick  gnatwick@harris.com 

ASEI-Authorized  Introduction  to  CMMI®  Instructor 
ASEI-Authorized  SCAMPI®''^  Class  A  Lead  Appraiser  (former) 
ASEI-Authorized  SCAMPI®''^  Class  B&C  Team  Leader  (former) 
AHarris  SEI  Partner  Business  &  Technical  Point  of  Contact 

Debra  Perry  dperrv@harris.com 


Det  Norsks  Veritas  (DNV)  Q-Labs 
16340  Park  Ten  Place,  Suite  100 
Houston,  Texas  770845-5143 


http://www.dnv.com/ 
SEI  Partner 


David  Card  david.card@dnv.com 

AAuthor  of  Measuring  Software  Design  Quality  (Prentice  Hall,  1990) 

ACo-author  of  Practical  Software  Measurement  (Addison  Wesley,  2002) 

ACo-editor  ISO/IEC  Standard  15939:  Software  Measurement  Process  (International  Organization  for 
Standardization,  2002) 

AEditor-in-Chief  of  the  Journal  of  Systems  and  Software 


Capability  Maturity  Model  Integration,  CMMI,  and  CMM  are  registered  with  the  U.S.  Patent  and  Trademark  Office. 

SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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A  Major  supplier  of  a  broad  range  of  products 
A  Major  subsystem  supplier 
A  Becoming  a  system  supplier  in: 

I  ISR 
T  Training 

V  Aircraft  modernization  and  O&M 
T  Government  services 

A  Major  provider  of  national  security  solutions  in: 
I  C4ISR 

V  Homeland  security  and  defense/GWOT* 

V  Government  enterprise  IT 

V  Transformational  programs 
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ie  IT  Solutions  (EITS) 

vision  Overview 


communications 


Enterprise  IT  Solutions 


A 

A 

A 

A 


A 


Organization:  Division  of  L-3  Communications 

Employees:  Over  2,000 
professionals 

Headquarters:  Reston,  VA 

Chartered  to  support  civil 
and  defense  Government 
agencies 

Mission:  Provide  world-class  enterprise 
information  technology  (IT),  communications, 
and  engineering  services  and  solutions  to 
the  public  sector. 


A  Vision:  Become  the  Government’s  trusted  partner  for  exceptional  IT, 
communications,  and  engineering  services  and  solutions;  and  achieve  a 
challenging  and  rewarding  work  environment. 
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A  EITS  Division  composed  of  diverse  business  units  operating  under 
multipie  industry  models  and  standards  (CMMI,  ISO  20000,  ITIL, 
PMBOK) 

A  Government  and  public  agency  customer  base 

T  NASA  (National  Air  and  Space  Administration)  -  IV&V  (independent 
verification  and  validation  services) ;  CMMI  ML  3  Objective 

I  Metropolitan  airport  authorities  (business  process  engineering)  CMMI  ML  3 
Objective 

I  County  School  Systems  (IT  infrastructure  and  support)  ISO  20000  Objective 
I  Federal  Government  (staff  augmentation)  CMMI  MLS  Objective 

I  FAA  (Federal  Aeronautics  Administration  software  development)  CMMI  ML  3 
Objective 

A  Many  (sometimes  very)  small  projects  in 

I  software  development  functional  area  (CMMI,  PMBOK)) 

I  managed  services  functional  area  (ISO  20000,  ITIL,  PMBOK) 

A  staff  augmentation  projects  predominate  (CMMI,  PMBOK) 
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A  EITS  measurement  program  must  efficiently 
support  CMMI,  ISO  20000  (ITIL),  PMBOK  best 
practices 

A  EITS  measurement  process  assets  must  be 
tailorable  to  diverse  functional  areas 
(managed  services,  staff  augmentation) 

A  EITS  measurement  activities  must  have 
minimum  impact  on  limited  project  staff 
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\  Customizing  measurement  solutions  for  non-homogenous 
business  and  functional  areas 

I  Selecting  the  nrightomeasurements  to  best  support  business 
goals 

T  Cost  effective  staffing  of  measurement  activities  in  small 
short  term  projects  with  minimal  resources 

T  Effective  monitoring  and  control  of  CMMI  process  areas  with 
minimal  measurement  resources 

I  Mapping  CMMI  model  measurement  best  practices  based 
on  arger  software  development  projects  into  small  non 
software  development  projects 

I  Integrating  and  reusing  measurements  based  on  CMMI 
measurement  practices  to  support  implementation  of  other 
industry  standards  (ITIL,  ISO  20000,  PMBOK) 
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Generic 

“CMMI  Guidelines  for  Process  Integration  and  Product  Improvement”  Second 
Edition;  Crissis,  Konrad,  Schrum  2006 


“Monitor  and  control  the  process  against  the  plan  for 
performing  the  process  and  take  appropriate 
corrective  action  .... 


Subpractice  1.  Measure  actual  performance  against  the 
plan  for  performing  the  process” 


Gill  IdiniBHIlM-IIHflll 
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The  Dilemma ...  BH 

Apparent  gaps  uncovered  during  CMMI  GP  2.8 
implementation  in  EITS  NASA  IV&V  projects 

A  Initial  expectation:  existing  IV&V  measurement  program 
adequately  covered  CMMI  measurement  requirements  with 
only  minor  gaps 

A  Reality  check:  generally  the  case  except  for  CMMI 
requirements  around  institutionalization  of  GP  2.8 

A  Concern:  measurements  would  need  to  be  implemented  in 
all  projects  being  appraised  for  all  process  areas  at  maturity 
levels  2  and  3  V  resulting  in  almost  30  new  measurements 
per  project! 


11/16/2007  Susanna  Schwab 


8 


^  Complete 


use  period  has  ended. 
Thank  you  for  using 


Your  compiimentary 


PDF  Compiete. 


The  Questions  ...  ? 


A  What  sort  of  measurements  are  appropriate  and  useful  to 
monitor  and  control  each  process  area? 

A  Are  measurements  necessary  for  each  process  area  being 
assessed? 

A  Are  there  alternative  qualitative  methods  to  monitor  and  control 
process  areas? 

A  How  do  projects  tailor  monitor  and  control  of  process  area 
quantitative  or  qualitative  activities? 

A  How  should  senior  management  be  informed  and  involved  with 
monitor  and  control  of  process  performance  in  projects? 

A  How  can  monitor  and  control  of  process  be  implemented  in  a 
time  and  cost  effective  manner? 
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The  Happy  Ending  .... 

A  BITS  division  +  iV&V  team  chartered  to  map  existing 
iV&V  measurement  to  generic  measurements  and 
address  any  gaps 


A  almost  3  months  of  contentious  discussion  ensued  in 


attempt  to  address  gaps  in  least  burdensome  manner 

A  qualitative  measurement  alternatives  suggested  for 
low  value  process  areas;  a  few  simple  to  collect  but 
useful  measurements  added 

A  solution  strategy  reviewed  and  approved 


CMMI  success  ! 
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1)  Use  qualitative  alternatives  to  measurement 
where  appropriate 

I  Strategically  use  qualitative  alternatives  to 
measurement  (where  appropriate)  to 
minimize  overhead 


Aka  K.I.S.S.  ^ 
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Build  on  the  KISS  principle 

>  CMMI  GP  2.8  requires  that  monitor  and  control  of  process 
areas  be  institutionalized. 

>  Obvious  mechanism  to  do  this  is  to  define  measurements 
for  each  process  area 

>  May  be  expensive,  time  consuming,  and  non  value  added 

>  Division  defines  suggested  measurements  for  each 
process  area  but 

>  Projects  identify  key  process  areas  for  measurement  and 
reporting  i'  other  process  areas  are  monitored  and 
controlled  qualitatively  with  reporting  by  exception 
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2)  Carefully  define  measurement  tailoring 
guidelines  and  validate  tailoring  execution 


Generic  division 

defined 

measurement 

Tailored  functional 
area  measurement 
or  alternative 

Collection  and 
analysis  role 

Reporting 
role  and 
frequency 

Actual  cost  compared 
to  budget 

Earned  Value  Cost 
Variance 

Project  Manager 

Project 

Manager 

Monthly 

Product  defects 

Number  of  formal 
customer  issues 

Functional  area 
Quality  System 
Manager 

Quality 

System 

Manager 

Quarterly 

Decision  Analysis 
Review  (DAR) 
scheduled  versus 
actual 

DAR  performance 
stoplight 

Functional  Area 

QA  auditor 

Quality 

System 

Manager 

Quarterly 
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Use  Generic  measurements  with  tailoring  validation 

>  Generic  measurements  for  process  area  monitoring  and 
control  specified  at  division  level  with  tailoring  guidelines 

>  Existing  project  measurements  mapped  to  generic 
specifications 

>  Minimal  set  of  additional  measurements  and  qualitative 
alternatives  identified,  reviewed,  approved  and  implemented 
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3)  Collect  and  analyze  measurements  at 
highest  possible  level  of  organization 


Measurement  implementation 


Measurement  institutionalization; 
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“Push  up”  implementation 


>  Collect  data  at  organizational  level  of  related  business  goal 


>  Measurements  supporting  division  goals  collected,  analyzed, 
and  reported  by  division  measurement  roles 

>  Measurements  supporting  functional  area  goals  collected, 
analyzed,  and  reported  by  functional  area  measurement 
roles 


>  Projects  collect  and  report  only  project  operational 
measurements 
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4)  Push  institutionalization  down  to  lowest 
organizational  levels 


Measurement  implementation 


Measurement  institutionalization; 
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>  Measurements  supporting  process  goals  for  common 
processes  collected,  analyzed,  and  reported  by  higher 
organizational  level  but  e 

>  Projects  collect  and  report  project  operational 
measurements 


>  Projects  receive  and  use  measurements  reported  by  all 
organizational  levels 
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5)  Leverage  organizational  measurement 
resources  and  best  practices 


Ctw|ui 


EITSiSoalsa 

Objectives 


□rive - ^ 


Policy  and 
Requirements 


Mapv'Supporl: 


FA  Goals  Sl 
ObJsctJvss 


EITS  MA  Strategic 
Framework  for  CMMI 
and  ISO  2000 


Drive 


EITS 

Measurement 

Handbook 


MapMSnpfJort 


I  t 

FA  MA  Plan 

I 

Tailor 


Map/Support 


Proiect  Goals  and 
Objectives 


-Drit 


Project  Plan 


-Define — > 


Tailor 
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Leveraging  organizational  assets  and  best  practices 

>  Division  develops  measurement  framework  (specifications, 
tailoring  guidance,  interfaces)  to  support  all  standards  and 
practices 

>  Functional  areas  develop  application  specific  measurement 
planning  frameworks  with  tailoring  guidance  ;  best  practices 
shared 

>  Projects  tailor  from  functional  area  measurement  planning 
framework;  best  practices  shared 
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Measurement  program  preparation  for  CMMI 
MLS  appraisal  of  NASA  IV&V  projects 

A  Generic  measurements  for  process  area  monitoring 
and  controi  specified  at  division  levei 

A  Existing  IV&V  measurements  mapped  to  generic 
measurements;  gaps  identified 

A  Division/IV&V  working  team  chartered  to  address 
gaps 

A  Minimal  set  of  additional  measurements  and 
qualitative  alternatives  identified,  reviewed,  approved 

and  implemented 
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^  Use  qualitative  alternatives  to  measurement 
where  appropriate 

^  Carefully  define  measurement  tailoring 
guidelines  and  validate  tailoring  execution 

^  Collect  and  analyze  measurements  at  highest 
possible  level  of  organization 

^  Push  institutionalization  down  to  lowest 
organizational  levels 

^  Leverage  organizational  measurement 
resources  and  best  practices 
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Measurement  Manager 
Enterprise  IT  Solutions  Division 
L-3  Communications 
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703-434-4796 

susanna.schwab@l-3com.com 
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12-15  November  2007 


nft  Rick  Hefner,  Ph.D. 

^  Director,  Process  Management 
Northrop  Grumman  Corporation 

rick.hefner@ngc.conn 


Background 


■  Software  measurement  remains  a  challenge  for  many 
projects  and  organizations 

■  It  is  difficult  to  select  a  set  of  measures  that  are  easy 
to  define  and  collect,  yet  offer  real  insight  into 
progress,  process,  and  quality 

■  This  presentation  will  discuss  strategies  for  starting 
and  enhancing  a  CMMI-compliant  measurement 
system 


NORTHROR  CRUMM/XN 

Rick  Hefner,  "Measurement  Strategies  in  the  CMMI",  24  April 

Copyright  2005  Northrop  Grumman  Corporation 


CMMI 

Measurement  and  Analysis  Process  Area 


■  Purpose 

■  Develop  and  sustain  a  measurement  capability  that  is  used  to 
support  management  information  needs 

■  Involves  specifying: 

■  Information  needs  and  measurement  objectives 

■  Measures 


■  Data  collection  and 


Practical  Software  Measurement 


storage  mechanisms 

■  Analysis  techniques 

■  Reporting  and  feedback 


ISO/IEC  15939^  Software  Measurement  Process 


mechanisms 


■  Written  to  conform  to 
ISO/IEC  15939, 

Software  Engineering  - 
Software  Measurement 
Process 


CMMI 

Measurement 
And  Analysis 


ISO/IEC  SC7  Standards 


12207  (revision  -  supporting  process) 
15288  (measurement  concepts) 
9126  (terminology  coordinated) 
14598  (terminology  coordinated) 

ISO  9000-3:2000  (objectives) 


NonmnoR  crummam 


Rick  Hefner,  "Measurement  Strategies  in  the  CMMI",  24  April  2C 


Copyright  2005  Northrop  Grumman  Corporation 


Practical  Software  and  Systems  Measurement 

Measurement  Principles 


■  Measurement  is  a 
consistent  but 
fiexibie  process  that 
must  be  taiiored  to 
the  unique 
information  needs 
and  characteristics 
of  the  project  or 
organization 

■  Decision  makers 
must  understand 
what  is  being 
measured  and  trust 
the  information 

■  Measurement  must 
be  used  to  be 
meaningfui 


Reference:  http://www.psmsc.com 


MORTHROR 
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Practical  Software  and  Systems  Measurement 

Multi-Level  Measurement  Requirements 


■  Different  types  of  information  are  needed  at  different  levels  of  the 
infrastructure 


*  Perfo/7]?3Jice  Measurement 

Enferpr/se  *  Normative  Performance  Baselines 

/Wan element  *  Jechnlcai  and  Business  Poitcy 

t  investment  Decisions  &  Anaiysis 


Organizational 

Management 


Process  improvement 
Project  Pianning  Guideiines 
Perfonnance  Based  Guideiines 
Organizational  Wojrjiis  &  Benchmarks 


*■  Project  Estimation  A  Planning 
Project  *  Project  Performance  Tracking 

/Wanagemenf  ^  Project  Tradeoff  Analysis 

t  Resource  Management 


informaiion  ■  Driven 
Measurement 
—  Process  ” 

\ 
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Practical  Software  and  Systems  Measurement 

Analysis  Model 


:$ 
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ISO/IEC  15  3  ,  Software  Engineering  - 
Software  Measurement  Process 


In  fbffustion 
Fruduct 


information  Needs 


Indfcafor 

Estimate  OF  Evaluation  that 
Provides  a  Basis  for  Decision 
Making 


Algorithm  Co-mbining  MEasurES  and 
JWodol  Decision  Criteria 


Derived' 

Derived 

ATeasure 

iWea-SLire 

Quantity  DEfined  as  a  Function  of 
Two  or  More  MeasurES 


leasoremenil  Algorithm!  Combining  Two  or  Mote 
Function  J  Base  Measures 


A  MEasurE  of  a  Single  Attribute 
By  a  Specific  Method 


Ease 

Ease 

iWea-sfijre 

JU  es'S  ore 

fIWeaSiJrieJ7?eof|  ^oasureirpenfl  Operations  Quantifying  an 
Mottiod  ^  Mefftod  j  AttributE  Against  a  Scale 


Affrfbute 

A  frribdjfe 

Property  Relevant  to 
Information  Needs 


Entities 
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CMMI 

Measurement  and  Analysis  Goal  1 


Goal/Practices 

Notes 

Typical  Evidence 

SG  1  Align  Measurement  and 
Analysis  Activities 

Measurement  objectives  and 
activities  are  aligned  with  identified 
information  needs  and  objectives. 

Focus  is  on  alignment  with 
objectives,  not  just 
specifying  a  set  of  metrics 

SP  1.1  Establish  Measurement 
Objectives 

Establish  and  maintain 
measurement  objectives  that  are 
derived  from  identified  information 
needs  and  objectives. 

See  following  slide 

Information 

needs 

Measuremen 
t  objectives 

SP  1.2  Specify  Measures 

Specify  measures  to  address  the 
measurement  objectives. 

List  of 
metrics, 
operational 
definitions 

SP  1.3  Specify  Data  Collection 
and  Storage  Procedures 

Specify  how  measurement  data  will 
be  obtained  and  stored. 

Collection 
and  storage 
procedures 

SP  1 .4  Specify  Analysis 

Procedures 

Specify  how  measurement  data  will 
be  analyzed  and  reported. 

Analysis 

procedures 
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Information  Needs  &  Measurement 
Objectives 


■  Information  needs  set  requirements  for  determining  the  needed 
metrics 

■  Measurement  objectives  set  requirements  for  determining  the 
needed  metrics  coilection,  storage,  anaiysis,  and  reporting 
mechanisms 


information  Needs 

What  types  of  information 
are  needed  by  the  project? 

■  Progress 

■  Quaiity 

■  information  needed  by 
the  organization 

■  information  needed  by 
the  customer 


Measurement  Objectives 

What  objectives  influence 
how  the  measures  are 
coliected,  analyzed,  stored, 
\  reported? 

/  ■  Accuracy 

■  Timeiiness 

■  Security 


MORTHROR  CHUMM/XN 
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Measurement  and  Analysis  Goal  2 


Goal/Practices 

Notes 

Typical  Evidence 

SG  2  Provide  Measurement 

Results 

Measurement  results  that  address 
identified  information  needs  and 
objectives  are  provided. 

Following  defined 
procedures 

SP  2.1  Collect  Measurement  Data 

Obtain  specified  measurement  data. 

Measuremen 
t  collection 
records 

SP  2.2  Analyze  Measurement 

Data 

Analyze  and  interpret  measurement 
data. 

Evidence  should  explicitly 
show  interpretations 

Analysis 

results 

Interpretation 

s 

SP  2.3  Store  Data  and  Results 

Manage  and  store  measurement 
data,  measurement  specifications, 
and  analysis  results. 

Data  storage 
records 

SP  2.4  Communicate  Results 

Report  results  of  measurement  and 
analysis  activities  to  all  relevant 
stakeholders. 

iVO/77WSF 

Metrics 

reports/ 

briefings 

rO/* 
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What  Does  the  Data  Mean? 


0) 

s 

o 

a 

E 

o 

u 

o 

a 

(A 

O 

£ 


Large  number  of  defects  found  in  high  complexity 
components;  will  require  second  review 


31  41 

Component 
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Management  Styles  in  the  CMMI 


Project 


Quantitative 

management 


Proactive 

management 


Reactive  mgmt. 
(plan,  track,  and 
correct) 


Level 

Process  Areas 

5  Optimizing 

Causal  Analysis  and  Resolution 
Organizational  Innovation  and  Deployment 

4  Quantitatively 
Managed 

Quantitative  Project  Management 
Organizational  Process  Performance 

3  Defined 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organizational  Process  Focus 

Organizational  Process  Definition 
Organizational  Training 

Risk  Management 

Integrated  Project  Management  (for  IPPD*) 
Integrated  Teaming* 

Integrated  Supplier  Management** 

Decision  Analysis  and  Resolution 
Organizational  Environment  for  Integration* 

2  Managed 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 
Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 
Configuration  Management 

1  Performed 

Organizational 


Quantitative 

improvement 


CRWfMAN 
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Measurement  at  CMMI  Level 


Organizational  Process  Performance 

■  Establishes  a  quantitative  understanding  of  the  performance  of  the 
organization’s  set  of  standard  processes 

■  Provides  process  performance  data,  baselines,  and  models  to  quantitatively 
manage  the  organization’s  projects 


measurement 

repository 


project  performance 


organizational 
standard  process 


project’s  defined 
process 


organizational 
performance  data 
&  models 


customer  and  project 
objectives 


Quantitative  Project  Management 

■  Quantitatively  manage  the  project’s  defined  process  to  achieve  the  project’s 
established  quality  and  process-performance 
objectives. 


NORTHROR  CRWIM/kN 
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Exercise 

What  is  Quantitative  Management? 


■  Suppose  your  project 
conducted  several  peer 
reviews  of  similar  code,  and 
analyzed  the  results 

■  Mean  =  7.8  defects/KSLOC 

■  +3a  =  1 1 .60  defects/KSLOC 

■  -3a  =  4.001  defects/KSLOC 


12  — 
11  — 
10  — 


CD 


0  5  10  15 

Observation  Number 


UCL=  11.60 


Mean=7.8 


LCL=4.001 


What  would  you 
expect  the  next  peer 
review  to  produce  in 
terms  of  defects/ 
KSLOC? 

What  would  you 
think  if  a  review 
resuited  in  10 
defects/KSLOC? 

3  defects/KSLOC? 


MOKWrmOR 
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Exercise 

What  is  Required  for  Quantitative  Management? 


■  What  is  needed  to  develop  the 
statistical  characterization  of 
a  process? 


12  — 
11  — 
10  — 


0) 


0  5  10  15 

Observation  Number 


UCL=  11.60 


Mean=7.8 


LCL=4.001 


The  process  has  to  be 
stable  (predictable) 

■  Process  must  be 
consistently  performed 

■  Complex  processes  may 
need  to  be  stratified 
(separated  into  simpler 
processes) 

There  has  to  be  enough 
data  points  to  statistically 
characterize  the  process 

■  Processes  must  occur 
frequently  within  a  similar 
context  (project  or 
organization) 


15 
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Typical  Choices  in  Industry 


Most  customers  care 
about: 


Defect  Detection  Profile 


■  Delivered  defects 

■  Cost  and  schedule 

So  organizations  try  to 
predict: 

■  Defects  found 
throughout  the  lifecycle 

■  Effectiveness  of  peer 
reviews,  testing 

■  Cost  achieved/actual 
(Cost  Performance 
Index -CPI) 

■  Schedule 
achieved/actual 
(Schedule  Performance 
Index -SPI) 


Req'mts  Design 


Code  Unit  Test 

Phase 


Integrate 


SysTest  Del  90  Days 


Process  performance 
•Process  measures  (e.g.,  effectiveness, 
efficiency,  speed) 

•Product  measures  (e.g.,  quality,  defect 
density). 


NORTHROR  GRUMMAN 
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Measurement  at  CMMI  Level  5 


■  Organizational  Innovation  &  Deployment 

■  Set  quantitative  improvement  goals  (e.g.,  reduce  variation  by  X%, 
reduce  mean  by  Y%) 

■  Seek  innovative  improvements  -  cause  a  shift  in  process  capability 

■  Analyze  potential  improvements  to  estimate  costs  and  impacts 
(benefits) 

■  Pilot  improvements  to  ensure  success 

■  Measure  the  impact  of  improvements  quantitatively  (variation  and 
mean) 


■  Causal  Analysis  &  Resolution 

■  Identify  and  analyze  causes  of  defects  and  other  problems 

■  Take  specific  actions  to  remove  the  causes  -  prevent  the 
occurrence  of  those  types  of  defects  and  problems  in  the  future 


Peer  Reviews  Improving  the  Process 


■  Reduce  the  variation 

■  Train  people  on  the  process 

■  Create  procedures/checklists  15 

■  Strengthen  process  audits 

■  Increase  the  effectiveness  § 

(increase  the  mean)  ^ 

■  Train  people  1  5 

■  Create  checklists 

■  Reduce  waste  and  re-work  “  0 

■  Replicate  best  practices  from 
other  projects 

0  5  10  15 

Observation  Number 


UCL=11.17 


Mean=7.268 


LCL=3.363 
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Lessons  Learned 


■  To  establish  (revitalize)  a  measurement  system,  start  by 
identifying  all  the  stakeholders  and  what  information  they  need  to 
make  decisions 

■  Look  for  common  needs,  which  drive  common  metrics  that  can  be 
used  by  many  stakeholders 

■  There  is  no  “magic”  set  of  metrics  that  works  for  every  project  or 
every  organization 

■  It  takes  several  months,  if  not  years,  to  develop  an  effective 
measurement  system 

■  Initially,  focus  is  on  ensuring  data  is  provided 

■  Next,  focus  in  on  data  definition  problems 

■  Finally,  focus  on  effective  use  of  the  data 

■  Concentrate  on  developing  a  data-driven  culture 


■  When  moving  to  Levels  4  and  5,  expect  a  period  of  trial-and-error 
to  discover  the  metrics  you  need 

■  Focus  on  management  by  variation  (e.g..  Six  Sigma) 
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^  Complete 
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PDF  Complete. 


rade  to 
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Scope  of  Events  Discussed 


>  3  very  large 
organizations  in  last 
three  years 

>  SE/SW 

>  SW  only 

>  SE/SW/IPPD 

>  Wide  array  of  types  of 
work.  2  Global,  1  U.S. 
Fed  and  State.  2  did 
not  have  external 
customer  CMMI  Level 
requirements.  Ail  deal 
with  multiple 
frameworks.  1  has 
been  doing  CMM 
based  improvement 
for  a  many  years. 
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^  Complete 
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PDF  Complete. 
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and  Expanded  Features 


What  is  an  Enterprise  Appraisal? 


An  event(s)  that  leads  to  a  ratable  Class  A  benchmark 
appraisal  that  includes  multiple  sub-units  which  in  and 
of  themselves  are  ratable  OUs. 

■  Includes  more  than  one  sub  unit. 

■  Includes  corporate  level  organization  units  (above  the  typical 
OU  scope) 


^  Why? 

■  Confirm  standard  process  roll  out  and  execution 

■  Gain  competitive  advantages 

■  Accept  and  work  with  reality  of  constant  organizational  changes 

^  Is  it  one  “big  honking  appraisal?”  -  No!! 


Copyright  ©  2007  ISO,  Inc. 


Experiences  Implementing  Very  Large,  High  Confidence  Enterprise  Appraisals 


3 
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Your  complimentary 


PDF  Complete. 
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and  Expanded  Features 


❖ 

❖ 

❖ 


Stakeholder  Concerns  to  Address 


Senior  sponsor:  fis  this  going  to  bust  our  budgets?  What®  the 
benefit?6 

Business  Unit  sponsor:  f1  don4  want  to  jeopardize  my  bonusio 

Program  Managers:  fWhy  do  I  care  about  these  of/7er  business 
units?6 

EPG  members:  fHow  can  I  support  ail  these  events  and  heip 
peopie  improve  toolo 

Enterprise  Lead  Appraiser:  fHow  do  I  ensure  that  ali  these 
appraisais  are  run  effectively  T  I  can4  be  on  them  allio 

Lead  Appraisers:  f1  dondwant  my  appraisai  at  risk  with  SEI  by 
doing  some  non-standard  evente  !6 

SEI:  (We  dondwant  any  SCAMPI  principles  violated  ot 
requirements  missed,  and  we  don^want  organizations  making 
crazy  level  claims !6 


❖ 

❖ 

❖ 
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PDF  Complete. 
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Several  Innovations  and  Improvements 


Org  (enterprise  level)  appraisal  elements 
Incremental  Data  Reuse 
Org  Sampling  criteria 

❖  PHD  Refresh  events  (practice  sampling) 
Verification  Reviews 

Strategic  Appraisal  Plan 
Central  Appraisal  Planning 
Implementation  fWaveso 
Common  tooling  (and  work  instructions) 
<$>  Common  training 

Common  Interpretations 
<$>  Norming  with  Leads 


We  will  discuss  these 
bullets  throughout  the 
presentation 
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PDF  Complete. 
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Appraisal  Goals  i  Enterprise  Impacts 


Common 

Goal 

Sub-Goal 

Enterprise  Appraisal 
Implementation 

Ensure  results 

Contribute  directly  to 
business  improvement 

Comparable  across 
companies/organizations 

Increased  specificity  needed 

Comparability  required 

Customer  “beiievability” 
essential 

Optimize  value  to 
sponsors 

Support  business 
objectives 

Optimize  cost  and  minimize 
disruption 

Muitipie  requirements  must  be 
satisfied 

Enterprise  “big  picture” iocus 

Ensure  appraisal 
reliability 

Create  repeatable 
processes  i  standardize 

Make  results  predictable 
and  differences  explainable 

Results  independent  of 
team  composition 

Objectivity  essential 

Use  of  external  (non-OU) 
resources  increases 

Standardization  needed. 

Slide  adapted  and  updated  from  presentations  by  Mr.  Byrnes  while  managing  the  appraisal  project  at  the  SEI. 
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Appraisal  Goals  i  Business  Unit  SCAMPIs 


^  Provide  a  thorough,  objective  benchmark  against  the  CMMI 

^  Baseline  the  process  capability  of  each  targeted  business  unit  against  the 
CMMI  V1.1,  Staged  Representation,  using  the  SCAMPI  V1.1  method 

^  Ensure  events  are  led,  managed,  and  executed  in  a  manner  that  is 

■  ARC  compliant, 

■  fully  defensible,  and 

■  results  are  acceptable  to  respective  clients  requiring  reference  model 
benchmarks. 

^  Ensure  each  entity  receives  appraisal  assets  that  are  usable  by  the 
business  unit  sponsor  /ndepenc/enf  of  any  final  Enterprise  ML  rating 

^  Receive  an  official  CMMI  Maturity  Level  Rating  from  a  team  led  by  an 
external  SEI  Authorized  SCAMPI  Lead  Appraiser 

<$>  Conduct  each  appraisal  within  schedules  tailored  in  each  appraisal  plan  to 
meet  overall  Enterprise  and  Business  Unit  specific  appraisal  objectives. 
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PDF  Complete. 
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and  Expanded  Features 


Scope  I  Enterprise  ORG 


Level 

Focus 

Process  Areas 

5  Optimizing 

Continuous 

Process 

Organizational  Innovation  and  Deployment 

Improvement 

Causal  Analysis  and  Resolution 

4  Quantitatively 

Quantitative 

Organizational  Process  Performance 

Managed 

Management 

Quantitative  Project  Management 

Quality 

and 

Efficiency 
are 

increasec 


3  Defined 

Enterprise  level  entities 
reviewed  separately  or  in 
conjunction  with 
underlying  unit  Class  A 
events. 


2  Managed 


Process 
Standardization 


Basic 

Project 

Management 


Requirements  Development 
Technical  Solution 
Product  Integration 
Verification 
Validation 


Organizational  Process  Focus 
Organizational  Process  Definition 
LOrganizational  Training! 


IntegratediProject  Management 

Risk  Management 

Decision  Analysis  and  Resolution 


Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 


1  Initial 


Risk  and 
Rework 
are 

Reduced 


Copyright  ©  2007  ISO,  Inc. 


Experiences  Implementing  Very  Large,  High  Confidence  Enterprise  Appraisals 


8 


s^PDF 

Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 
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and  Expanded  Features 


Reference  Model  Scope  i  Overall 


Target  Process 
Capability 

Rating  Baseline 

Rating  Elements 

Other  Data 

reuse 

(For  each  sub-unit 
SCAMPI) 

CMMI V 1 . 1  Levels  x 
and  y,  Staged 
Representation 

Full  Scope,  Full 

Coverage  with  formal 
ratings  of  all  Level  x 
and  y  PAs 

Maturity  Level  rating 
required 

Joint  ISD/client  team 

Maturity  Level 

Process  Areas 

Process  Area  Goals 
Generic  practices 

Specific  practices 

Results  of  underlying  business  unit 
benchmark  Class  A  appraisals  and 
the  Enterprise  level  risk  appraisal 
(Class  B)  event  and  document 
review  performed  during  the 
Readiness  Review  may  be  re¬ 
used,  as  applicable,  within  the 
teama  appraisal  database. 

(For  enterprise  event) 
CMMIvl.l  Staged 
Representation, 
Organization  process 
areas 

Full  coverage  with 
process  area  ratings 
for  Organization  level 
Process  Areas  (OPD, 
OPF,  OT,  OEI) 

Process  Areas 

Process  Area  Goals 
Generic  practices 

Specific  practices 

Practice 

Samplinq 

A  sampling  of  practice 
implementation  across  prior 
appraised  units  will  be  re¬ 
validated  as  part  of  the  Enterprise 
appraisal  to  ensure  continued 
institutionalization  of  sub  unit 
ratings  [called  PHD  refresh 
events]. 

(For  each  sub-unit 
SCAMPI) 

(For  enterprise  event) 
CMMIvl.l,  Staged 

None 

None 

Resulting  appraisal  artifacts  from 
underlying  SCAMPI  Class  A 
predecessor  events  will  be 
verified  by  the  Enterprise  Lead 
Appraiser  for  ARC  compliance. 
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Org  Scope  i  3  Primary  Event  Types 


Company 

Business  Unit 

Location 

Site  visit  dates 

ivel 

I 

<Very  Large  Company 

X> 

<Named>  Sector 
<Named>  Organizational 
Units 

Multiple  locations 
throughout  the 
United  States. 

Multiple  throughout  <several 
years> 

Many  sub 
unit  Class 

A’s 

<Very  Large  Company 

X> 

<Enterprise  Organization 
entity> 

<On  site  City, 

State> 

<on  site  peric  Enterprise  Le 
“0”  appraisa 

<Very  Large  Company 

X> 

<Named>  Sector 

Some  <Named> 
Organizational  Units 
[PHD  refresh  events] 

Varied 

<On  Site  Period>  and  other  dates 
within  3  months  of  enterprise 
SCAMPI 

PHD  Refresh 
events 
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Organization  Scope  i  Enterprise  SCAMPI 

For  the  enterprise  level  SCAMPI,  the  Organizational 
infrastructure  entities  appraised  in  entirety  or  in  part: 

Senior  Leadership 

Enterprise  Process  Group  (EPG) 

Quality  Management  and  Delivery  Assurance 

^  Human  Resources 

Organizational  Training 

Knowledge  Management  (infrastructure  and  tools) 

Entities  in  large  organizations  typically  above  the  division  level  that  create, 
deploy,  and  maintain  common  assets  across  the  whole  enterprise. 
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Enterprise  Appraisal  Results 


^  The  Enterprise  SCAMPI  Class  A  event  results  in 

■  Process  Area  ratings  for  CPF,  OPD,  and  OT  for  organization  entities 

■  An  overall  Enterprise  Maturity  Level  rating  based  on  the  combined 
results  of  the  Enterprise  SCAMPI  and  the  results  of  each  underlying 
Wave  1  Business  Unit  SCAMPI  Class  A. 

The  Enterprise  SCAMPI  Class  A  event  does  not  re-benchmark 
underlying  business  unit  SCAMPI  results. 

■  Each  sub-unit  has  been  rated  separa^e/y  with  full  coverage  and  its  own 
ADS 

■  Where  appropriate  (ratings  outside  90  day  Enterprise  event  window), 
PHD  Refresh  events  are  conducted  to  confirm  capabilities  are  still  in 
place. 

■  Business  Unit  Class  A  appraisal  assets  and  results  are  verified  to 
ensure  adequacy,  completeness,  and  ARC  compliance. 
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Appraisal  Considerations 


Appraisal  practices 
(examples) 

Implementation  issues,  risks,  and 
recommendations 

Appraisal 

considerations 

Plan  the  Process  (GP 

2.2) 

Organizations  often  donQ  know  how 
much  data  is  needed  relative  to  prior 
events  when  increasing  scope. 

Must  engage  outside  Lead  sooner.  Do 

Central  Appraisal  Planning. 

Sampling  strategies  need  to  be  documented. 
Align  goals  across  units,  not  just  within. 

Use  historical  appraisal  data  for  estimating. 

Identify  and  Involve 
Stakeholders  (GP  2.7) 

Very  broad  set  of  stakeholders.  Easy  to 
miss  key  people.  May  involve  groups  not 
previously  part  of  appraisals. 

When  nnewO  groups  involved,  they  exhibit 
Plow  appraisal  maturity 6  despite  organization 
overall  process  capability. 

More  prep  time  needed. 

Do  training  even  if  they  already  had  it. 

Establish  a  Defined 
Process  (GP  3.1) 

Organizations  often  focus  on  procedures 
within  processes,  rather  than  with 
interfaces,  coordination,  synergy,  and 
integration  across. 

Need  documents  that  describe  connections 
across  process  elements  and  organizational 
boundaries. 

Review  Status  with 
Higher  Level 
Management  (GP  2.10) 

Many  issues  and  decisions  can  be  driven 
down  to  lower  levels  appraisals. 

Manage  the  effort  like  a  project.  Decompose 
the  problem.  Track  metrics.  Set  norms  up 
front. 

Manage  Configurations 
(GP  2.6) 

Data  across  company  in  multiple 
repositories.  Significant  IT,  security  and 
archival  concerns  and  needs. 

Need  for  good  CM  to  manage  incremental 
appraisal  database  build  up  and  reuse  over 
several  events.  IT  infrastructure  critical. 
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Addressing  Risks 


Risk 

Factors 

Actions 

Maintaining 
senior  mgt. 
commitment 

Caused  by  turnover  or  mergers 

Caused  by  management  changes 

Issues  resulting  from  shifting  investment  priorities 

Dual  or  tri-sponsorship  for 
major  events  i  enterprise, 
EPG,  Business  Unit  T 
interfaces  established 

Middle  mgt. 
resistance 

Overriding  pressure  for  project  performance; 
Incentives  on  delivery,  not  quality 

Focus  on  Level  rather  than  improvement 

Assign  EPG  TPOCs  for 
each  unit. 

Minimize  disruption. 

Inappropriate  or 
conflicting  goals 

Focus  on  Level  rather  than  improvement 

Business  Unit  Level  x  goal,  Enterprise  level  y  goal 

Ensure  each  major  sub 
unit  is  intervened  with. 

Tailor  events  V  not  force 
single  approach. 

Unrealistic 

expectations 

All  OU®  benchmarked  by  year  end  in  the  4^*^ 
quarter. 

Start  Up  projects  Level  x  by  year  end. 

Spread  events  over  long 
period.  Establish 
incremental  strategy  and 
roll  up.  Define  fwave 
strategy. 

Crash 

implementations 

PHD  mania. 

Big  bang  appraisals. 

Process  in  a  box. 

Lots  of  efforts  on  going  at 
any  one  time. 

Not  one  hmegaoeffort. 
Several  methods  in  tool  kit. 
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and  Expanded  Features 

Risk  Management  Activities 


^  Spend  extra  time  up  front  defining  the  organization  scope, 

strategy,  approach,  and  techniques.  How  much  time?  Years!  (this 
is  not  a  tacticai  effort!) 


<$>  Integrate  outcomes  from  a  series  of  events  for  each  business  u 
(swim  lanes).  Affinitize  units  into  fwaves,6for  deployment  and 
benchmarking,  (this  is  practicai!) 

<$>  Standardize  appraisal  assets  for  use  by  a  commonly  trained  set 
of  appraisers,  using  a  central  appraisal  planner,  (these  are 
essentiai  and  sometimes  iearned  after  the  fact!) 


<$>  Norm  the  set  of  Leads  T  each  Lead®  ways  of  doing  business  on  a 
one-off  needed  to  adjust  slightly,  (this  is  chaiienging!) 


^  Involve  the  SEI  throughout,  at  key  pivot  points  (this  was  hard!) 
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iger  B’s,  smaller  A’s,  appraisal  lifecycle  model 

Expanded  Features 

Example  Appraisal  Team  Set  Up 


Appraisal 

Event 

Team  Size 

Days  on  site 

Team  Comp.: 
External  -  External 
to  OU  -  Internal  to 
OU 

Effort  hours 
/Team 
Member 
(normative) 

At  ieast  2  totaiiy 

Trends 

Class  A 

6-8 

5-7 

external,  Vz  non 
OU 

45 

i 

Class  B 

6-8 

7-10 

Tried  to  have 
same  team  as  A 

64 

t 

Class  C 

1-3 

3-5 

Usuaiiy  internal 
or  expert  driven 

24 

Readiness 

Review 

4  or  more 

5 

V2-1 .0  size  of  A 

40 

t 

PiiD 

Refresh 

4 

3-4 

Vz  the  size  of  A; 
ali  from  A  team 

24 

1 
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What®  a  Wave? 


Due  to  size  and  complexity  of  the  organization,  processes  and 
process  improvement  activities  can  be  deployed  in  fwaves.6 

■  Mechanism  to  prioritize  EPG  involvement 

■  Mechanism  to  focus  organization  improvement  where  end  customer  or 
project  specific  needs  are  most  pressing 

■  Establishes  and  exceeds  reasonable  percentages  for  organization 
coverage  for  enterprise  and  separate  business  unit  ratings 

■  Accounts  for  reality  that  not  all  programs  will  be  at  same  maturity  state 
at  same  time 

■  Ensures  process  deployment  across  entire  Enterprise 

■  Reduce  risk,  increase  success  rate,  manage  complexity 

<$>  Assumption:  Not  all  units  targeted  will  be  at  the  same  stage  of 
maturity,  or  readiness  for  change,  or  ability  to  implement  changes. 
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just  the  high  level  flow  -  there  was  much  more  detail 


Conceptual  Diagram  i  Deployment  fWaveso 


FY06 


FY07 


FY08 


FY09 


3A 


4A 

4B 

4C 


5A 

5B 

5C 


Months  01 0203040506070809  10  11  12010203040506070809  10  11  12010203040506070809101112010203 

Initial  Global  L3 


1 

Lx  by  <date> 

1A 

06 

1A 

IB 

07 

IB 

2 

Lx-1  by  TBD 

U[ 

2A 

07 

2A 

2B 

07 

2B  1 

3 

Lx+  by  TBD 

07 


3A 


07  Progi 

08 


4A 


Lx+  by  TBD 


07 

08 

08 


4B 


5A 


L3+ 


■4C  L 


5B 


Significant  time  spent  with 
Enterprise  Lead  discussing 
sampling  appropriateness 


3B 

07 

3B  \  \ 

3C 

08 

3C 

4 

Lx-1  by  TBD  07  nfw 

Update  Global  L3 


5C 


Notional  timing  for  discussion  and  illustration  purposes  only 
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What®  a  PHD  Refresh  Event? 


Purpose:  Verify  process  still  in  place  and  implemented  for  a 
previously  benchmarked  unit. 

Need:  Cancb  realistically  perform  all  required  Class  A  events  in  a  90 
day  pre-Enterprise  Class  A  window  due  to 

■  Business  unit  specific  needs  and  objectives 

■  Resource  constraints 

■  Practical  project  work  flow  issues 

^  Timing:  performed  within  90  day  window  of  Enterprise  Class  A 

Timing  criteria  relative  to  last  successful  Class  A  benchmark 

■  0-3  months:  use  underlying  data  as  is  T  full  reuse 

■  4-12  months:  do  PHD  refresh  to  confirm  current  status 

■  >12  months:  do  full  Class  A  event 
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is  is  just  the  high  level  criteria  -  there  was  much  more 


PHD  Refresh  Guidelines/Criteria 


Environmental  Attribute 


Current  State  Relative  to 
Benchmark  Event 


Risk  to  Incremental 
Appraisal 
Outcomes 


Risk  Mitigation 
Activities 


Major  re-organizations 


None/List  specific  change, 
date,  and  impact 


Low/Medium/High 


<Describe  actions 
taken> 


Major  acquisitions 

Major  changes  in  standard 
process 

Significant  changes  in 
plans/scope  of 
appraised  projects 


Senior  Management  changes 


Organization  restructuring 


Process  implementation 
changes 
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was  an  entire  Appendix  and  an  embedded  document 
Strategic  Appraisai  Rian  dedicated  to  this  topic. 


What®  a  Practice  Sampling  Plan? 


Purpose:  Tailor  follow  on  appraisal  event  to  minimize  cost  and 
disruption  on  an  organization  that  has  already  successfully  executed 
a  full  Class  A  but  must  participate  in  the  Enterprise  event. 


❖  Approach:  Obtain  maximum  actual  OE  coverage  through  optimizing 
a  tailored  set  of  practices  reviewed.  Pick  fheavy  hitteroand 
frepetitiveopractices.  Use  precedence  and  dependency  relationships 
inherent  in  the  model.  Example: 


Level 

Process 

Area 

Goal 

Practice 

E 

L 

o 

u 

Decision  Criteria  Rationale 

2 

REQM 

SP  1.3 

X 

X 

Need  to  be  able  to  manage  changes  and  reconcile  project  issues  as  they  change  and 

SP  1.5 

X 

ensure  all  relevant  assets  are  getting  updated. 

GP2.6 

X 

Making  sure  controlling  requirements  key. 

GP2.8 

Ensure  Org  level  is  collecting  requirements  metrics. 

2 

PP 

SP  1.2 

X 

X 

Estimates  always  an  issue. 

SP  2.7 

X 

Plan  updates  affect  ever5Thing  else  and  will  see  the  other  goal  2  practices. 

SP  3.2 

X 

Reconciling  tasks/resources  always  an  on-going  challenge. 

GP2.6 

X 

Controlling  changes  to  plans,  estimates,  etc.  tends  to  be  a  typical  issue  area. 

GP2.2 

Ensure  org  level  is  getting  plans  from  programs 
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What®  an  Asset  Verification  Review? 


^  Purpose:  Ensure  all  underlying  events  leading  to  the 
Enterprise  SCAMPI  Class  A  event  were  performed  with 
high  quality  and  in  accordance  with  all  SCAMPI 
requirements. 

Approach:  Develop  and  use  a  standard  appraisal 
requirements  checklist  to  perform  reviews  of  all  key 
appraisal  deliverables  for  each  event 

■  Plans,  briefings,  reports,  ADS,  etc. 

■  Document  issues,  recommendations  and  gaps  as  ftindingsofor 
corrective  action. 

■  Issues  in  underlying  events  could  potentially  delay  the  final 
Enterprise  outcome 
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Class  A  Requirements  Checklist  -  Sample 


Activity 


Task 


Requirements 


Verification  Notes 


Verified 


Analyze  Requirements 


Determine  Appraisal 
Objectives 


Identify  Sponsor  and  Relevant 
Stakeholders 


In  Strategic  Appraisal  Plan 
Section  2.0 


Document  Business  and 

Appraisal  Objectives 


In  Strategic  Appraisal  Plan_ 


Alignment  of  Appraisal 
Objectives  with 
Business  Objectives 


In  Strategic  Appraisal  Plan 
Section  3.0  and  in 
Team  In  Brief  and 
Organization  In  Brief 


Determine  and  Document 
Appraisal  Usage 
Mode 


In  Strategic  Appraisal  Plan 
throughout  and  in 
Team  In  Brief 
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Objective  Evidence  Challenges 


Need  common  rules  and  guidance  as  to  instantiations  required. 

^  Need  work  instructions  on 

■  how  to  present  data, 

■  how  much  data  is  needed,  and 

■  how  the  team  is  to  record  its  review  of  the  data. 

Need  for  automated  tools  increased  T  expansion  in  data 
data  reuse  strategy,  merging  of  data  increases  need  for  different 
approaches  to  recording  data 

<$>  Organization  Coverage:  large  units  have  a  real  challenge  of 

showing  institutionalization  across  the  entity  when  only  reviewing  a 
small  set  of  projects  in  a  Class  A  T  how  many  instances  is  enough? 
What  percentage  of  the  unit  is  enough? 

<$>  Functional  Coverage:  there  may  be  “org”  groups  that  need  to  be 
covered  at  multiple  layers  of  the  overall  enterprise  (corporate, 
division,  business  unit). 
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Model  Interpretation  Issues 


^  What  is  the  norgofor  OPD,  OPF,  and  OT  purposes? 

<$>  What  makes  up  the  rratableometrics  repository? 

^  How  ffconnectedomust  the  enterprise  be  to  the  units? 
And  vice  versa? 

Team  needs  ability  to  nntegrateorather  than  de¬ 
compose  [holistic  perspective]  for  the  Enterprise  event. 
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Pitfalls  and  Take  Aways 

^  Pitfalls 

■  Dond  assume  everyone  will  understand  on  the  first  run. 

■  All  sub-units  must  buy  into  the  approach  as  well,  even  if  they  have 
some  specialized  unit  appraisal  objectives. 

■  Appraisal  experience  matters. 

■  Team  members  that  have  worked  with  each  other  before  matters. 

■  Work  instructions  matter. 

<$>  Take  Aways 

■  Management  support  is  really  needed. 

■  Communication  vehicles  must  be  routinely  delivered. 

■  Standard  assets  and  common  training  facilitate  easier  comparisons. 

■  Central  planning  helps  ensure  consistency 

■  IT  infrastructure  for  evidence  collection,  asset  archive  repository,  and 
team  activities  is  essential. 
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Some 
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Key  Organizational/Appraisal  Challenges 

Organizational 

■  Too  many  models.  Too  many  methods. 

■  Management  drivers  for  reduced  process  improvement  costs. 

■  Need  to  increase  efficiency  of  both  internal  improvement 
activities  and  external  appraisal  efforts. 

■  Customer  ndisconnectsobetween  fievel  achievementoand 
fproject  performance.6 

Appraisal 

■  Data  element  needs  increase  and  morph  with  enterprise  focus 

■  Some  SCAMPI  rules  can  actually  get  in  the  way 

■  Changes  in  method  not  fast  enough  to  keep  up  with  changes  in 
organizational  needs 
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Issues,  Directions,  and  Opportunities 


Issues 

#  SCAMPI  V1 .2  still  focused  on  rsingle  point 
appraisals, onot  set  of  integrated  appraisals 
from  an  enterprise  perspective 

Directions 

#  Starting  second  wave  on  2  major  accounts. 
^Improvements  in  approach  being  documented 
now. 

^Continue  technical  development 

Opportunities 

#  Technical  approaches  taken  were  considered 
a  great  success  from  all  key  stakeholders: 
Sponsors,  EPG  lead.  Enterprise  Lead  Appraiser 

#  Interface  with  SEI  for  potential  updates  to 
SCAMPI 
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Questions  and  Answers 
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Contact  Information 


^  Paul  D.  Byrnes,  pdbvrnes(a)isd-inc.com 
^  Integrated  System  Framework  (ISF). 

■  http://www.isd-inc.com 

■  Follow  links  technical  presentations 


Integrated  System  Diagnostics,  Inc. 

Massachusetts  Corporate  Headquarters 
889  Shore  Road 
Post  Office  Box  3440 
Pocasset,  MA 
Tel:  (508)  564-5626 
http://www.isd-inc.com 
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Background 


alternative  practice  -  A  practice  that  is  a  substitute  for  one  or  more 
generic  or  specific  practices  contained  in  CMMI  models  that  achieves 
an  equivalent  effect  toward  satisfying  the  generic  or  specific  goal 
associated  with  model  practices.  Alternative  practices  are  not 
necessarily  one-for-one  replacements  for  the  generic  or  specific 
practices. 

--  Glossary,  CMMI  for  Development  Version  1.2 


■  What  does  this  mean? 

■  Under  what  conditions  do  alternative  practices  occur? 

■  How  do  you  judge  whether  they  are  acceptable? 
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Understanding  the  Context  of  the  CMMI 


■  context  -  1 :  the  parts  of  a  discourse  that  surround  a  word  or  passage 
and  can  throw  light  on  its  meaning;  2:  the  interrelated  conditions  in 
which  something  exists  of  occurs 

-  Merriam-Webster  OnLine  Dictionary 


■  CMMI  is  a  best  practice  modei 

■  It  reflects  best  practices  that  address  development  and 
maintenance  activities  applied  to  products  and  services 


■  What  is  “best”  in  a  given  situation  (i.e.,  a  deveiopment  activity) 
depends  on  the  context 
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An  Example  of  Context 


■  “You  should  not  talk  with  your  mouth  full?” 

■  This  is  a  best  practice  -  a  good  general  rule  to  be 
followed 


■  Are  there  contexts  in  which  the  rule  doesn’t  apply? 
What  if: 

Your  toddler  is  about  to  touch  a  hot  stove? 

You’re  demonstrating 

why  talking  with  your  mouth  full  looks  bad? 

The  culture  considers  talking  with  your  mouth  full 

proper  and  polite? 
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How  Does  this  Apply  to  CM  MI? 


The  structure  of  the  CMMI  is: 

■  Goals  are  appropriate  in  any  context  envisioned  by  the 
CMMI  authors 

■  Hence,  they  are  required; 

■  Practices  are  appropriate  in  most  contexts 

■  Hence,  are  expected 

■  Alternative  practices  may  be  appropriate  in  the  other 
contexts; 

■  Subpractices,  etc.  are  appropriate  in  some  contexts 

■  Hence,  are  treated  as  informative 

■  Because  in  many  contexts  they  may  not  be  appropriate. 
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What  is  the  Context  Assumed  by  the  CMMI 
Authors? 


■  There  is  no  explicit  statement  of  the  assumed  context 
(e.g.,  large  DoD  contractor,  small  commercial 
company,  etc.)  for  any  practice 

■  Each  author  was  probably  biased  by  the  types  of 
examples  they  had  seen  in  their  own  organization 

■  Also,  the  same  context  is  not  assumed  for  all 
informative  material  throughout  the  model 

■  Different  authors,  different  times  =  different  contexts 

■  Hence,  the  informative  material  is  simply  one  example 
of  a  myriad  of  ways  that  might  be  appropriate  for 
meeting  the  practices,  not  the  only  way,  or  even  a 
preferred  way 
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An  Example  Level  /5 


■  At  the  time  CMMI  was  written,  most  industry  exampies 
were  software  organizations  that  repeatediy  deveiop 
the  same  type  of  software 

■  Similar  programming  languages,  similar  applications, 
similar  staff,  similar  project  goals 

■  Quite  a  different  context  than  a  geographicaiiy- 
distributed  US  DoD  contractor  with  a  wide  dispersion 
of  project  types  implementing  a  Six  Sigma 
methodology 

■  Result  --  Some  informative  material  in  QPM  assumes 
projects  quantitatively  manage  the  same 
subprocesses  quantitatively  managed  in  OPP 
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The  Definitions  Provide  Ciues  as  to  Context 


^  ■  project  -  a  managed  set  of  interrelated  resources  which 
^  delivers  one  or  more  products  to  a  customer  or  end  user. 


A  project  has  a  definite  beginning  (i.e.,  project  startup)  and 
typically  operates  according  to  a  plan...  A  project  can  be 
composed  of  projects. 

■  How  does  this  definition  fit  your  scope  of  work? 

■  Contracts  with  many  different  deiiverabies 

■  Programs  composed  of  multiple  projects 

■  Maintenance  work 

■  Service  projects 
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ATLAS  10  Survey  Structure 


■  Candidate  alternative  practices  were  solicited  from  the 
community  at  large;  requested  submission  of  either: 

■  Practices  actually  implemented;  or 

■  Ways  of  describing  “alternative  practices” 


■  77  respondents  -  44  unique  candidates  were  submitted 

■  44  candidates  consolidated  into  11  groups  of  four 

■  Each  group  was  distributed  randomly  to  the  SEI- 
authorized  individuals 
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ATLAS  10  Question  1 


Please  select  the  letter  that  best  represents  your  view  of  this 
candidate  alternative  practice 

A.  I  strongly  agree  [that  this  an  acceptable  alternative  practice] 

B.  I  somewhat  agree  [...] 

C.  I  neither  agree  nor  disagree  [...] 

D.  I  somewhat  disagree  [...] 

E.  I  strongiy  disagree  [...] 

■  Each  response  (A-E)  for  each  candidate  aiternative  practice  was 
quantified  as  foliows: 

■  A  or  B  (I  strongly/somewhat  agree):  +1  point 

■  C  (I  neither  agree  nor  disagree):  0  points 

■  D  or  E  (I  somewhat/strongiy  disagree):  - 1  point 

■  A  candidate  alternative  practice’s  “score”  =  the  average  across 
ail  respondents.  For  the  44  candidate  alternative  practices: 

■  Score  Range:  +0.59  to  -0.85 

■  Score  Mean:  -0.25 

■  Score  Median:  -0.26 
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Example  1:  SAM  SP  1.2  (Score:  0.5  ) 


SP  1.2  Select  Suppliers 

Select  suppliers  based  on  an  evaluation  of  their  ability  to  meet 
the  specified  requirements  and  established  criteria. 

■  Rather  than  selecting  a  supplier,  our  org  has  the  suppliers 
imposed  by  our  primary  customer. 

■  The  ability  of  the  suppiier  to  meet  the  requirements  is  analyzed, 
and  the  results  of  this  anaiysis  are  presented  to  the  customer.  If 
there  are  concerns  about  the  supplier’s  ability  to  meet  the 
specified  requirements,  risks  are  documented  and  shared  with 
the  customer,  or  managed  internaliy  by  the  org. 

■  Experience  logs  are  maintained  for  each  suppiier  to  influence  the 
customer’s  supplier  seiection  in  the  future. 

■  The  direct  artifacts  for  this  candidate  aiternative  practice  are  the 
notification  from  the  customer  that  we  must  use  the  designated 
suppiier,  the  analysis  report,  and  associated  risks,  and  the 
experience  iogs  maintained  for  each  supplier. 
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How  Do  We  Determine  Whether  This  is  an 
Acceptable  Alternative  Practice? 


alternative  practice  -  A  practice  that  is  a  substitute  for  one  or  more 
generic  or  specific  practices  contained  in  CMMI  models  that  achieves 
an  equivalent  effect  toward  satisfying  the  generic  or  specific  goal 
associated  with  model  practices. 

SP  1.2  Select  Suppliers 

Select  suppliers  based  on  an  evaluation  of  their  ability  to  meet 
the  specified  requirements  and  established  criteria. 


SG  1  Establish  Supplier  Agreements 

Agreements  with  the  suppliers  are  established  and  maintained. 


■  What  effect  are  we  trying  to  achieve? 


■  What  would  an  equivalent  effect? 
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Is  the  Informative  Material  Helpful  in 
Judging  Acceptability? 


Criteria  should  be  established  to  address  factors  that  are 
important  to  the  project. 

Examples  of  factors  include  the  following: 

•  Geographical  location  of  the  supplier 

•  Supplier’s  performance  records  on  similar  work 

•  Engineering  capabilities 

•  Staff  and  facilities  available  to  perform  the  work 

•  Prior  experience  in  similar  applications 
Typical  Work  Products 

1 .  Market  studies 

2.  List  of  candidate  suppliers 

3.  Preferred  supplier  list 

4.  Trade  study  or  other  record  of  evaluation  criteria, 
advantages  and  disadvantages  of  candidate  suppliers,  and 
rationale  for  selection  of  suppliers 

5.  Solicitation  materials  and  requirements 

Subpractices 

1 .  Establish  and  document  criteria  for  evaluating  potential 
suppliers. 

2.  Identify  potential  suppliers  and  distribute  solicitation 
material  and  requirements  to  them. 

A  proactive  manner  of  performing  this  activity  is  to  conduct 
market  research  to  identify  potential  sources  of  candidate 
products  to  be  acquired,  including  candidates  from 
suppliers  of  custom-made  products  and  vendors  of  COTS 
products. 


5.  Evaluate  proposed  suppliers'  ability  to  perform  the  work. 

Examples  of  methods  to  evaluate  the  proposed  supplier’s 
ability  to  perform  the  work  include  the  following: 

•  Evaluation  of  prior  experience  in  similar  applications 

•  Evaluation  of  prior  performance  on  similar  work 

•  Evaluation  of  management  capabilities 

•  Capability  evaluations 

•  Evaluation  of  staff  available  to  perform  the  work 

•  Evaluation  of  available  facilities  and  resources 

•  Evaluation  of  the  project’s  ability  to  work  with  the 
proposed  supplier 

•  Evaluation  of  the  impact  of  candidate  COTS  products  on 
the  project's  plan  and  commitments 

When  COTS  products  are  being  evaluated  consider  the 
following: 

•  Cost  of  the  COTS  products 

•  Cost  and  effort  to  incorporate  the  COTS  products  into  the 
project 

•  Security  requirements 

•  Benefits  and  impacts  that  may  result  from  future  product 
releases 

Future  releases  of  the  COTS  product  may  provide 
additional  features  that  support  planned  or  anticipated 
enhancements  for  the  project,  but  may  result  in  the 
supplier  discontinuing  support  of  its  current  release. 

6.  [Select  the  supplier. 


3.  Evaluate  proposals  according  to  evaluation  criteria. 


4.  Evaluate  risks  associated  with  each  proposed  supplier.. 
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So  How  Prevalent  are  Alternative  Practices? 


■  Only  5  of  the  44  submitted  candidates  had  more 
authorized  individuals  supporting  the  assertion  that 
they  were  true  alternative  practices  than  refuting  it 


■  That  is,  only  5  candidate  alternative  practices  had  a 
score  >  0. 

■  Given  that  5  did  pass  a  relatively  simple  litmus  test,  it 
may  be  concluded  that  “alternative  practices”  are 
REAL,  and  NOT  merely  conceptual! 

■  However,  given  that  ali  44  were  submitted  as  viable 
candidates,  it  appears  that  “alternative  practices”  are 
not  interpreted  consistently  across  the  population  of 
authorized  individuals 
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ATLAS  10  Question  2 


If  you  selected  either  D  or  E  above  (i.e.,  the  candidate  is 

unacceptable),  please  indicate  your  rationaie: 

A.  The  candidate  is  not  sufficientiy  different  from  the  modei  practice 
to  be  considered  an  “alternative” 

B.  Although  an  “alternative,”  it  does  not  appear  to  support  goal 
satisfaction  as  well  as  the  practice  as  written 

C.  It  is  not  acceptable  because  it  eliminates  the  practice  without 
providing  a  viable  alternative 

D.  Other 


■  Although  most  respondents  that  found  a  candidate 
alternative  practice  unacceptable  did  provide  a 
response  to  Item  #2,  the  choice  (A  -  D)  did  not  aiways 
align  with  the  supporting  comments 

■  Bottom  line:  Little  useful  insight  was  gleaned  from 
analyzing  the  responses  to  Item  #2 
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ATLAS  10  Question  3 


Regardless  of  its  alternative  practice  candidacy,  assuming  that 
there  are  ample  direct  artifacts  supporting  consistent  practice 
implementation  on  all  projects  as  indicated,  please  provide  your 
“gut-feel-characterization”  for  <practice>  (considering  the 
organization  and  projects  as  described). 

_  (FI,  LI,  PI,  Nl) 
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ATLAS  10  Question  3  Responses 


Some  candidate  alternative 
practices  experienced 
significantly  more 
variation  than  others 


Candidate 


10 


12 


13 


14 


19 


24 


27 


28 


32 


34 


25 


26 


44 
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Moving  Forward 


■  In  the  final  analysis,  aiternative  practices  are  rare 

■  The  context  assumed  by  the  authors  (and  reviewers)  is  very 
broad,  (e.g.,  small/big  projects,  small/big  organizations, 
defense/  commercial,  different  business  goals) 

■  Many  purported  “alternative  practices”  are  better  described 
as  “alternative  implementations” 

■  Some  purported  “alternative  practice”  can  be  an  attempt  to 
avoid  changing  an  existing  process 

■  In  identifying  legitimate  alternative  practices,  look  for 
differences  in  the  assumed  context 

■  Definitions  of  “project”,  “organization”,  “customer” 

■  Verbs  which  are  not  possible  actions  in  your  context,  e.g., 
“select” 

■  Even  “experts”  disagree  about  the  acceptabiiity  of  an 
alternative  practice  (or  the  adequacy  of  its  implementation) 

■  Discuss  all  alternative  practices  with  your  Lead  Appraiser 
before  the  aoDraisal 
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Backup  Slide 

Example  2:  PMC  SP  1.7  (Score:  0.3  ) 


SP  1.7  Conduct  Milestone  Reviews 

Review  the  accomplishments  and  results  of  the  project  at 

selected  project  milestones. 


■  Our  org  does  not  develop  “traditional”  projects  but  does 
maintenance  work  using  time-boxing.  Our  management 
conducts  monthly  meetings  with  our  customers  to  measure 
progress,  assess  risks  and  determine  whether  the  features  to  be 
included  in  the  next  release  are  satisfactory  or  not. 


■  This  is  not  a  milestone  meeting  as  it  is  not  event-driven.  Because 
of  the  large  number  of  minor  enhancement  projects,  it  was 
decided  that  this  was  a  better  approach  than  trying  to  have  “real” 
milestone  meetings  on  every  enhancement.  There  are  typically  5- 
6  such  monthly  meetings  per  release. 


■  The  direct  artifacts  for  this  candidate  alternative  practice  are  the 
minutes  from  the  customer  meetings  as  well  as  the  documented 
issues  and  action  items  resulting  from  them. 
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Backup  Slide 

Example  3:  CM  SP  1.2  (Score:  0.25) 


SP  1.2  Establish  a  Configuration  Management  System 
Establish  and  maintain  a  configuration  management  and  change 
management  system  for  controlling  work  products. 


■  We  only  have  one  customer  for  whom  we  develop  and  support 
software  products.  Our  org  is  contractually  required  to  use  our 
customer’s  CM  and  change  management  control  (CMC)  systems. 
We  have  no  need  to  establish  and  maintain  a  CM  or  CMC  system 
of  our  own,  and  rely  solely  on  our  customer’s  systems  to  protect 
our  configuration  items  and  change  requests. 

■  The  direct  artifacts  for  this  candidate  alternative  practice  are  the 
customer’s  CM  and  CMC  systems  -  and  a  demo  of  how  we 
maintain  our  configuration  items  and  change  information  using 
these  systems. 
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Backup  Slide 

Example  :  VAL  (Score:  0.25) 


■  Our  government  customers  require  the  system  to  be  validated 
prior  to  acceptance.  However,  they  require  this  to  be  done  under 
their  control  using  their  validation  environment,  procedures,  and 
users. 

■  Since  we  can’t  deem  Validation  to  be  “not  applicable”  and  still  be 
rated  MLS,  we  have  decided  instead  to  treat  this  as  an  alternative 
practice. 

■  The  direct  artifacts  for  this  alternative  practice  are  the  customer 
contract  dictating  how  validation  is  to  be  performed,  and  the 
customer-run  validation  test  results. 
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Contact  Information 
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rick.hefner@ngc.com 


Patrick  O’Toole 

Process  Assessment, 
Consulting,  &  Training 
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Background 


The  SCAMPI  method  has  significant  flexibility  and 
tailoring  options 

Unfortunately,  many  Lead  Appraisers  do  not  take 
advantage  of  these  options 

■  Some  continue  to  conduct  appraisals  in  the  same  style 
as  the  discovery-based  CBA 
years  ago 


PI  methods  used  over  10 


This  presentation  discusses  the  fundamental  value- 
added  steps  of  a  SCAMPI  appraisal,  and  how  to  tailor 
the  methods  to  different  organizational  situations 

■  Preparation  (scoping,  planning,  evidence  gathering) 

■  On-site  (evidence  review,  interviews,  consolidation) 

■  Close-out  (reporting,  record  keeping) 
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Topics 


■  Understanding  the  purpose  of  a  SCAMPI  appraisal 

■  Identifying  the  non-value  added  appraisal  activities 

■  Scoping  and  planning  the  appraisal  for  minimum  cost 

■  Tailoring  choices,  and  how  to  make  them 

■  Preparing  the  evidence 

■  Eliminating  known  time-wasters 

■  Being  a  smart  buyer 
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Characteristics  of  CM  MI  Appraisal  Classes 


#'l  ■  S< 


M 
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The  ARC  (Appraisal  Requirements  for  CMMI)  defines  appraisal  classes 

A  guide  to  inventors  of  appraisal  methods,  and  their  customers 

SCAMPI  is  a  family  of  ARC-compliant  methods 

Appraisal  Requirements  for  CMMI,  Version  1.1,  CMU/SEI-2001-TR-034 


Characteristics 

Class  A 

Class  B 

Class  C 

Amount  of  Objective  Evi¬ 
dence  Gathered  (relative) 

Medhmi 

Low 

Ratings  Generated 

No 

No 

Resource  Needs  [relative) 

High 

Medium 

Low 

Team  Size  (relative) 

Lai'^e 

Medium 

Smalt 

Appraisal  Team  Leader 
Requirements 

Lead  appraisei' 

Lead  appiais^ei'  or 
person  traii’.ed 
and  exper  ienced 

Persci:  tiained 
and  experienced 

SCAMPI-A 


SCAMPI-B 


SCAMPI-C 


“A  Quantitative  Comparison  of  SCAMPi  A,  B,  and  C,  ”  R.  Hefner  and  D.  Luttreii, 
CMMi  Technoiogy  Conference  and  User  Group,  2005 


MonmmR  Grumman 


Hefner,  "Cutting  Appraisal  Costs  in  Half,  2007 


Copyright  2005  Northrop  Grumman  Corporation 


4 


A  Variety  of  Appraisals 


“Lower  Cost,  More  Effective  Alternatives  to  SCAMPIs,  ”  R.  Hefner,  2007  CMMI  Technology 
Conference  and  User  Group,  Thursday,  Nov  15,  3:30  pm 
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Applying  Six  Sigma  To  Appraisais 


Several  Six  Sigma  projects 
were  conducted  to  optimize 
the  SCAMPI  appraisal  process 


45 


40 


35 


30 

25 


S  20 


15 


10 


Evidence 
Prepari 


Evidence 


Interviews  Review 

Consolidation 


“Minimizing  SCAMPi  Costs  via  Quantitative 
Methods,  “  R.  Hefner  and  Ron  Uirich,  CMMi 
Technoiogy  Conference  &  User  Group,  17-20 
November  2003 


Collected  metrics  on  time  spent 
on  various  appraisal  activities, 
defects 


Used  Pareto  chart  to  identify 
bottlenecks,  opportunities  for 
improvement 


Used  individuals  charts  to  study 
variation  in  the  appraisal  process 


Used  fishbone  charts  and  other 
causal  analysis  methods  to 
identify  potential  improvements 


Key  considerations: 

■  Project  preparation  time 

■  On-site  appraisal  time 

■  Cost  &  resources 

■  Accuracy  of  appraisal  results 
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Mapping  the  Process  to  Identify  Bottlenecks 


Suppliers 

Inputs 

Assessment 

Schedules 

Process 

Outputs 

Improvement 

Plan 

Customers 

Organizations 

Projects 

Project  Personnel 

Project 

Evidence 

Savings 

Appraisal 

Teams 

Lead  Appraisers 

Personnel 

Availability 

SCAMPI 

Appraisals 

Effective 

Appraisals 

Organizations 

Project 

Schedules 

Lead  Appraisers 

Process  Steps 


Hefner,  "Cutting  Appraisal  Costs  in  Half",  2007 


NonmnoR  crummam 


Copyright  2005  Northrop  Grumman  Corporation 


Techniques  for  Reducing  Cost  -  Preparation 


■  Scoping  -  Determining  the  portion  of  the  organization  to  be 
appraised  (the  “organizational  unit”) 

■  Any  logical  portion  of  the  organization  may  be  chosen, 
e.g.,  a  division,  a  site,  a  domain,  etc. 

■  The  scope  wili  impact  both  the  utiiity  of  the  appraisai  results  in 
marketing  and  the  organizationai  buy-in 

■  :”Cherry-picking”  only  part  of  the  organization  to  be  appraised  may 
send  the  signai  that  CMMi  is  cost  without  vaiue 

■  Planning  -  Determining  the  budget,  schedule,  and  logistics 

■  Highly  driven  by  the  approach  to  evidence  review  and  interviewing 

■  Evidence  gathering  -  Compiling  the  direct  and  indirect  evidence 
needed  to  provide  compliance  with  the  CMMI  goals  and  practices 

■  Biggest  preparation  cost  and  effort 

■  Perceived  by  the  projects  to  be  non-value-added 
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Minimum  Team  Size 


■  Cost  is  composed  of: 

■  Team  costs  -  goes  up  with  team  members 

■  Organizational  costs  (interview,  presentations) 
-  largely  fixed  regardless  of  size 


■  Accuracy  goes  up  with  as  team  size  increases 


■  Buy-in  is  driven  by  the  confidence  the 
organization’s  members  has  in  the  appraisal 
process  and  appraisal  team 

■  Larger  teams  can  increase  the  likelihood  that  a 
respected  person  is  on  the  team 
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Team  Accuracy  vs.  Team  Size 


"  Same,  assuming  90%  leader 
accuracy 


As  team  size  goes  up,  team 
accuracy  rapidly  increases 
(assuming  the  right  answer  is 
obvious  once  presented) 

Teams  of  greater  than  4  provide 
littie  increase  in  accuracy 


If  the  team  leader  is  90% 
accurate,  additionai  team 
members  add  little  accuracy 

Adding  team  members  does  give 
a  chance  for  them  to  learn 


Appraiser  accuracy,  not  team  size,  is  criticai 

NORTHHOR  GRUMMAN 
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Evidence  Mapping  Should  Use  An  Automated 
Tool 


■  Key  Tool  Capabilities 

■  Point  to  existing  project 
fiie  structures 

■  Capture  status  and 
needed  actions 

■  Provide  statistics  over  time 
-  project  compiiance, 
organizational  compliance 


■  Identify  common  gaps  across  projects 


■  Identify  typical  evidence  for  each  practice 


Tips 

■  Finding  the  “right”  evidence  will  involve  iteration 

■  Remember  that  the  goal  is  improvement  (learning/implementing 
new  practices  effectively),  not  finding/creating  the  evidence 

■  Use  workshops  to  educate,  motivate,  populate 

■  Careful  preparation  reduces  on-site  evidence  review  time 
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Techniques  for  Reducing  Cost  On-Site 


■  Evidence  review  -  Evaluating  the  gathered  evidence  to  verify 
CMMI  goal  and  practice  compliance 

■  Remember  the  goal  is  to  validate  that  the  practice  is  performed,  not 
to  judge  goodness  of  the  document 

■  Inexperienced  appraisers  should  be  coached  to  develop  the  proper 
perspective  and  speed 

■  Interviews  -  Verifying  the  evidence  is  appropriate 

■  Not  as  important  as  evidence  review 

■  Simply  verifies  that  what  you  saw  is  what  is  being  used 
(verification,  not  discovery) 

■  Not  a  test  of  practioners’  memory 

■  Consolidation  -  Using  direct,  indirect  and  affirmations  to  form 
judgments  about  goal  and  practices  compliance 

■  Biggest  time-waster 
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Reducing  Interview  Costs 


■  Use  pre-scripted  interview  questions 

■  Conduct  interviews  simultaneously  in  mini-teams 
(Remember  that  more  than  3-4  people  don’t  increase 
accuracy  much.) 

■  Schedule  one  interview  per  practice  &  instantiation  (no 
SCAMPI  requirement  for  multiple  interview  sources  like 
in  CBA  IPI) 
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Reducing  Variation  in  Evidence  Review 


inexperienced 


The  time  is  takes  to  review 
evidence  is  predictable 

■  Some  variation  by  process 
area 

The  mean  review  time  and 
variation  is  much  higher  among 
inexperienced  appraisers 

■  At  least  half  of  the  appraisers 
on  the  team  should  be 
experienced 

Review  time  is  driven  by  the 
ciarity  with  which  evidence  is 
assembled  and  mapped  to  the 
CMMI  practices 

■  Ensure  thorough  evidence 
scrub  prior  to  on-site  period 


■  Inappropriate  evidence 
(“defects”)  causes  unexpected 
schedule  overruns 
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Reducing  Consolidation  Time 


Crafting  observations 

•  Voice  of  Customer  data  indicates 
organizations  and  projects 
simpiy  want  to  know  which 
practices  they  do  not  comply  with 

■  Consistent  with  Verification 
mode 

■  No  need  to  wordsmith  charts 

"  Use  an  Appraisal  Findings  tooi  to 
capture  the  ratings  at  the 
instantiation  ievei  (every  project, 
every  practice) 

■  Simplifies  data  consolidation, 
team  discussion 


Reviewing  as  a  team 

•  Most  of  the  time  is  spent  arguing 
about  how  to  interpret  a  few 
CMMI  practices 

■  Especially  Generic  Practices 

"  We  created  “CMMI  Interpretation” 
training  which  clarifies  how 
ambiguous  practices  wiil  be 
evaiuated 

■  Driven  by  areas  where 
disagreement  occurred 

■  Useful  in  reaching  team  (and 
organizational)  consensus 
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Ten  Most  Misinterpreted  CMMI  Practices 


"  Requirements  Management 

SP  1 .4  Maintain  Bidirectional  Traceability  of  Requirements 

■  Project  Planning 

SP  1 .2  Establish  Estimates  of  Work  Product  and  Task  Attributes 

■  Project  Monitoring  and  Control 

SP  1.1  Monitor  Project  Planning  Parameters 

"  Measurement  and  Analysis 

SP  1.1  Establish  Measurement  Objectives 

"  Configuration  Management 

SP  3.2  Perform  Configuration  Audits 

■  Verification 

SP  2.2  Conduct  Peer  Reviews 
SP  2.3  Analyze  Peer  Review  Data 

"  Risk  Management 

SP  1 .1  Determine  Risk  Sources  and  Categories 
SP  1 .3  Establish  a  Risk  Management  Strategy 

■  Generic  Practices 


“The  10  Most  Commonly  Misunderstood  CMMI  Practices,  “  R.  Hefner,  CMMI  Technology  Conference 
and  User  Group,  17-20  November  2003 

“Applying  CMMF  Generic  Practices  with  Good  Judgment,  “  R.  Hefner  and  G.  Draper,  CMMI 
Technology  Conference  and  User  Group,  15-18  November  2004 
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Summary 


■  Mission  Systems  is  typically  conducting  Level  5 
SCAMPI  appraisals  of  5-6  focus  projects  in  5-6  days 

■  Post-appraisal  follow-up  indicates  >95%  accuracy  rate 


We  are  continuing  to  look  at  ways  to  decrease  cost 
and  increase  effectiveness  and  value 

■  Effective  sampling  using  non-focus  projects 

■  Re-appraisals  to  prevent  “back-sliding” 

■  Handling  evidence  refresh 

■  Combining  with  ISO  9000,  AS-9100  appraisals 
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networks 


Aviation  electronics 


Intelligence,  surveillance,  and  Space  and  ground  satellite  Operations  and  support  services 

reconnaissance  communications  systems 
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T  Product-centric  approach 
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T  CMMI®  using  SCAMP|sm  qi^ss  A/B/C  appraisals 

Ensure  each  expected  artifact  description  is  clear 
and  complete  to  explain  why  it  is  relevant 

Maximize  the  re-use  of  actual  artifacts  to 
minimize  the  number  of  unique  artifacts 

Limit  the  impact  to  the  programs  by  minimizing 
the  changes 
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Organizational-centric  set  of  integrated  processes 
i'  Integrated  Process  Manual  (IPM) 
i'  Compliance  mapping  to  CMMI® 

Collaboration  across  functional  organizations 

Repeatable  processes  with  objective  criteria 
\  Entry/exit  criteria,  inputs,  outputs,  verification,  measures 
Planning  each  process,  and  tracking  against  plan 
i'  Tailoring  standard  processes  and  assets 
Budgets,  schedules,  resources 
Managing  established  baselines 
Managing  Stakeholder  involvement 
Measuring  progress  and  improvement 
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Programs  are  required  to  demonstrate  compliance  to  the 
organization®  integrated  processes,  as  defined  in  IPM 

PCM  tool  is  used  to  collect  artifacts  (i.e.  work  products) 

T  Each  process  statement  has  one  or  more  expected  artifacts 

V  Short  description  of  each  expected  artifact  provided 

T  Program  provides  work  product  name  and  location  that  meets 
that  expected  artifact  description 

A  PCM  tool  provides  objective,  online  auditing  and  real¬ 
time  monitoring  of  process  compliance 

T  QA  conducts  regular  assessments  of  the  artifacts  to  determine 
program  compliance  with  IPM 

T  Compliance  scores  are  recorded  in  the  tool 

AAvailable  to  the  team  and  management  in  real-time 

A  Reported  monthly  to  division  management 
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Overview 

A  brief  description  of  the  process  intent 


Entry  Criteria 

State,  Prerequisites,  Criteria 


Exit  Criteria 

State,  Criteria 


inputs  Outputs 

Needed  work  products,  resources  Resulting  work  products 


Required  Activities 

Mandatory  tasks  to  implement  the  process 


Measures 

Process  performance  against  plans 


Organizational  Improvement  Information 

Metrics,  reusable  work  products 


Verification 

Process  compliance  oversight 


Tailoring 

Approved  tailoring,  process  specific 

Implementation  Guidance 

Common  implementation  descriptions 

Supporting  Documentation  and  Assets 

Applicable  organizational  references 


Program  artifacts  needed 
to  demonstrate  I  PM 
process  compliance 
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A  Instead  of  looking  from  the  process  viewV 

looked  from  a  program  work  products  view 

A  Basic  guidelines 

T  Every  CMMI®  practice  shall  have  a  minimum  set  of 
adequate  expected  artifacts  in  PCM 

T  Every  IPM  statement  shall  have  a  minimum  set  of 
adequate  expected  artifacts  in  PCM 

T  Every  PCM  artifact  (existing  or  new)  shall  map  to  one 
or  more  IPM  statements  and  CMMI®  practices 

T  Maximize  the  re-use  of  existing  artifacts 
A  PCM  Startup  Template 
A  Standard  Directory  Structure 
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Mapped  program  work  products  to  I  PM  statements  and 
to  relevant  CM  Ml®  practices 

V  IPM  mapping  clearly  documented  in  PCM  tool 

V  CMMI®  mapping  in  PCM  tool  -  transparent  to  the  program 

A  Artifact  descriptions  clarified  to  help  the  program 
understand  relevance 

T  Descriptions  let  the  program  know  why  this  artifact  is  important 

T  IPM  perspective 

T  CMMI®  perspective 

A  Provided  name  of  typical  project  work  product  to  be  used 
as  an  artifact 

A  Provided  standard  directory  structure  location  where  that 
work  product  should  be  maintained 
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>ctory  Structure 


Supports  IPM  Compliance  with  artifacts  in  a  common 
structure  across  programs 

Top  level  directories  are  used  as  location  for  program 
artifacts 

V  Avoids  tying  PCM  artifacts  to  low  level  directories 
T  Easy  access  by  all  program  team  members 

V  Avoids  confusion  as  to  which  is  the  latest  version  of  an  artifact 

V  Flexibility  for  custom  directories  which  contain  fwork-in-progresso 

A  Pre-populated  with  latest  forms,  checklists  and  plan 
templates 

T  Set  up  by  IT  group  when  program  data  server  is  assigned 
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(^Contracts 
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^  Progrann_Controls 

File  Folder 

4/3/2007  1:04  PM 

(^  Progrann_Management 

File  Folder 

1/19/2006  12:10  PM 
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File  Folder 
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File  Folder 
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File  Folder 
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Owner.txt 

1  KB  Text  Document 

4/29/2007  5:09  PM 
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Work  products  reused  to  support  multiple  process 
statements 

T  Artifact  descriptions  provide  the  specific  appiication 

V  Minimized  the  number  of  unique  work  products  that  programs 
need  to  provide  in  PCM  tooi 

A  Tool  repositories  hold  many  of  the  program  artifacts 

T  DOORS,  ClearOuest,  Rose,  Pro-E,  etc. 

A  Some  evidence/artifacts  for  a  program  may  be  subject  to 
customer  data  requirements 

T  Programs  can  tailor  or  change  the  expected  artifacts  to  better 
align  with  their  execution 

V  Still  required  to  comply  with  the  IPM  (and  consequently  CMMI®) 
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A  Significant  reduction  in  the  number  of  artifacts 
needed  to  demonstrate  IPM  compliance 

I  Model-centric  approach 
A 1360  unique  artifacts 
I  Product-centric  approach 
A 326  unique  artifacts 
A7I8  pre-defined  artifact  descriptions 

Complete  mapping  to  CMMI®  practices  simplifies 
effort  required  for  SCAMPIsm  preparation 
I  Multiple  artifacts  map  to  CMMI®  practices 
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A  SCAMPISM  Class  C 
T  Planning 
T  Preparation 
T  Data  Review 

A  SCAMPISM  Findings 

T  Implementation  Risk 
T  Process  Definition  Characterizations 
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Given  three  different  sets  of  data  develop  a  map  to  show 
the  IPM  to  CMMI®  relationships 

V  IPM  statements 
i'  CMMI®  practices 
i  IPM/CMMI®  artifacts 

Capture  a  set  of  findings  to  characterize  the  process 
implementation  risks  and  degree  of  process  definition  for 
each  CMMI®  practice 

Make  the  task  of  preparing  for  and  conducting  an 
appraisal  as  simple  as  possible 
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GCSD  INTEGRATED 
PROCESS  MANUAL 


An  interim  appraisal  of  process  activities  to  revalidate 
existing  processes  based  command  media  against 
CMMn'DEV+IPPD  v1.2 

Context:  Command  media  recently  updated  to  reflect 
changes  in  the  organization®  process  improvement 
goals.  Desire  to  revalidate  existing  capability  with 
respect  to  CMMFi  DEV+IPPD  v1 .2 


Appraisal  Objective:  Conduct  a  SCAMPI^^  C  on  the 
GSCD  command  media  (documentation  only)  using 
CMMFl  DEV+IPPD  v1. 2 


Desired  Outcome  :  Provide  information  that 
management  can  use  to  baseline  process  performance 
and  to  prioritize  improvement  actions 
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Parent  Element 


When  loaded  into  the  tool  the  map  made  it 
easy  to  see  the  IPM  to  CMMI  relationships. 
This  allowed  us  to  simultaneously  review  the 
data  from  both  an  organizational  process 
need  (PCM)  and  a  model  (CMMI)  perspective 
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conduct  analyses  of  the  requirements 
with  the  requirements  provider  to 


Team  Members 


Data  Sources 


nKZ 

Otmi 

□  tmz 

□  tt^s 

□  ■^4 

□  "ms 

□  ■me 

□  ■m? 

□  ttis 


□  DPG/EPG  Interview 


Process  Corripliance  the  Smart  Way 


assure^  communications^ 


■ 


SEI  Partner 


NDIA  CMMI®  Conference  -  24 
12-15  November  2007 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


ngs 


hf/u^ms 


A  Compared  the  required  data  (as  defined  in  the  IPM)  to 
that  needed  to  satisfy  the  model 

A  Adjusted  the  total  dataset  as  needed  to  correctly  reflect 
artifacts  as  direct  and  indirect  evidence  or  to  remap  them 
if  mapping  errors  were  found 

A  Team  consensus  on  the  necessity  of  each  artifact  to 
demonstrate  complete  implementation  of  a  practice 

A  Concise  set  of  summary  findings  statements  to  reflect  the 
adequacy  of  the  data  set  and  potential  risk  of  successful 
deployment  and  implementation 
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As  the  project  matures  and 
requirements  are  derived,  all  acmities 
or  disciplines  TiiiU  receive  requirements. 

To  a’v’oid  requirements  creep,  criteria 
are  established  to  designate 
appropriate  channels,  or  official 
sources,  from  which  to  receive 
requirements.  The  recehing  acmities 
conduct  analyses  of  the  requirements 
with  the  requirements  provider  to 
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ClearQuest  {11696} 


Defined  criteria/checklists  for  evaluation 
and  acceptance  of  requirements  {11695} 


SRR  Materials  {90005} 


The  IPM  related  artifacts  were  reviewed  by 
the  team  to  determine  their  validity  as 
indirect  and  direct  evidence  for  each 
specific  and  generic  practice  of  the  CMMI. 

on  program  schedule 


Requirements  action 


H -482-5  to  ensure  that  the  customer  and  Harris  have  a  common 


Review  requirements  for  each  component  to  ensure  a  clear 
understanding  consistent  with  the  requirements  stakeholders. 


SRR  Materials  {90004} 


Requirements  specifications  {10003} 


►►N  +  -k 
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n^ue^iimU 


IPM 


Records  of 
requirements  reviews 


Review  requirements  for  each  component  to  ensure  a  clear 
understanding  consistent  with  the  requirements  stakeholders. 
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understanding  consistent  with  the  requirements  stakeholders. 
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clarifications,  rationale,  and  assumptions. 
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or  to  ensure  a  dear  understanding  consistent  with  the  requirements 


Define  the  specific  system  components,  work  products  and 
i  list  I  processes  that  will  be  validated,  and  the  validation  approach  to  be 
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Optional 


Signature  approval  of 
requirements 


Review  requirements  for  each  component  to  ensure  a  clear 
understanding  consistent  with  the  requirements  stakeholders. 


Accepted  [modify] 


Record  requirements  in  the  requirements  database,  including 
clarifications,  rationale,  and  assumptions. 


I  requirements  peer  understanding  consistent  with  the  requirements  stakeholders. 
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Fully  Defined 
(FD) 

- 1 - 

One  or  more  direct  artifacts  are  present  and  judged  to  be  adequate 

At  least  one  indirect  artifact  exists 

No  weaknesses  are  noted 

Largely 

Defined  (LD) 

One  or  more  direct  artifacts  are  present  and  judged  to  be  adequate 

At  least  one  indirect  artifact  exists 

One  or  more  weaknesses  are  noted 

Partially 

Defined  (PD) 

Direct  artifacts  are  absent  or  are  judged  to  be  inadequate 

One  or  more  indirect  artifacts  suggest  that  some  aspects  of  the 
practice  are  defined 

One  or  more  weaknesses  are  noted 

-OR- 

One  or  more  direct  artifacts  are  present  and  judged  to  be  adequate 

No  other  evidence  (indirect  artifacts)  supports  the  direct  artifact(s) 

One  or  more  weaknesses  are  noted 

Not  Defined 
(ND) 

Direct  artifacts  are  absent  or  judged  to  be  inadequate 

No  indirect  artifacts  support  the  practice  implementation 

One  or  more  weaknesses  are  noted 
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Harris  GCSD  (Defined  Process/Artifacts  by  Practice)  -  Apr  2007 


1  Not  Defined 

Partially  Defined 

1  1  Largely  Defined 

1  Fully  Defined 

REQM  PP  PMC  SAM  MA  PPQA  CM 
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Process  Areas  (PA) 
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Note:  Weaknesses  subsequently  mitigated  to  achieve  Fully  Defined 
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Label  Meaning 


Red 


The  intent  of  the  model  practice  is  judged  to  be  absent  or 
poorly  addressed  in  the  set  of  artifacts  identified  -  gaps  or 
issues  that  will  prevent  goal  achievement,  if  the  deployment 
occurred  in  this  way  across  the  organizational  unit,  were 
identified. 

The  intent  of  the  model  practice  is  Judged  to  be  partially 
addressed  in  the  set  of  artifacts  -  some  gaps  or  issues  were 
identified,  which  might  threaten  goal  achievement  if  the 
deployment  occurred  in  this  way  across  the  organizational 
unit. 

The  intent  of  the  model  practice  is  Judged  to  be  adequately 
addressed  in  the  set  of  artifacts  identified  -  in  a  manner  that 
would  support  goal  achievement,  if  the  practice  were 
deployed  across  the  organizational  unit. 
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A  Product-centric  approach 

T  Practical  and  proven  to  applying  across  organizational 
and  CMMI®  process  areas  and  practices 

T  Efficient  project  data  collection 

T  Fewer  redundant  findings 

I  Improved  support  for  projects  and  the  organization 

I  Maintains  integrity  of  the  appraisal  method  and 
achievement  of  sponsor  objectives 
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Harris  Corporation 
P.O.  Box  37 

Melbourne,  Florida  32902-0037 


http://www.harris.com/ 
SEI  Partner 


Gary  Natwick  gnatwick@harris.com 

ASEI-Authorized  Introduction  to  CMMI®  Instructor 
ASEI-Authorized  SCAMPI®''^  Class  A  Lead  Appraiser  (former) 
ASEI-Authorized  SCAMPI®''^  Class  B&C  Team  Leader  (former) 
A  Harris  SEI  Partner  Business  &  Technical  Point  of  Contact 

Dean  Wooley  dwoolev@harris.com 

AMember  of  American  Society  for  Quality  (ASQ) 

AISO-9001  internal  auditor 

AAppraisal  Team  Member  in  SCAMPI®"^  Class  A&C 


Integrated  System  Diagnostics,  Inc. 
780  South  Apollo  Boulevard,  Suite  107 
Melbourne,  FL  32901 


http://www.isd-inc.com/ 
SEI  Partner 


Jack  Lawrence  jlawrence(Sisd-inc.com 

ASEI-authorized  Introduction  to  CMMI®  Instructor 
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Scope  of  Events  Discussed 


>  5  Level  4-5  SCAMPI  A 
appraisals  over  last  3 
years 

>  SE/SW  (integrated) 

>  SE/SW  (separate 
ratings) 

>  SE/SW/SS 

>  SE/SW/IPPD/SS 

>  All  achieved  their 
desired  target.  One 
exceeded  their  target. 
One  was  a  re¬ 
appraisal. 

>  Roughly  one  third  of 
the  organizations 
providing  data  to  the 
SEI  for  their  latest 
“benefits”  report  are 
ISO  clients. 


>  Paul  Byrnes  has 
performed  app.  30 
Class  B  and  Class  A 
equivalent 
appraisals  for  CMM 
and  CMMI  since 
1996. 


>  ISO  currently  has  10 
certified  HM  Lead 
Appraisers  as  staff, 
affiliated 
consultants,  and 
teaming  partners  out 
of  123  total  as  of 
11/12/07. 
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Less  Process 


Areas  DoesnQ  Mean  Less  Effort! 


4  more 

Process  Areas 
at  Levels  4-5 
doesn’t  mean 
only  22% 
more  effort!! 

Heed  the  SEI 
published  data 
on  time  to 
move  up 
maturity 
levels! 

Going  from 
Level  3  to  4  in 
less  than  a 
year  would 
require  special 
cause  analysis. 


Level 

Focus 

Process  Areas 

5  Optimizing 

Continuous 

Process 

Improvement 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

Quality 

Productivity 

4  Quantitatively 
Managed 

Quantitative 

Management 

Organizational  Process  Performance 

Quantitative  Project  Management 

/ 

3  Defined 

Process 

Standardization 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organizational  Process  Focus 

Organizational  Process  Definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

/ 

2  Managed 

Basic 

Project 

Management 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

Risk 

1  Initial 

Rework 
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Common  Goals  i  High  Maturity  Impacts 


Common 

Goal 

Sub-Goal 

High  Maturity 
Appraisais 

Ensure  results 

Contribute  directly  to 
business  improvement 

Comparable  across 
companies/organizations 

Increased  specificity  required 

Integrating  with  other 
assessments  desired 

Optimize  value  to 
sponsors 

Support  business 
objectives 

Multiple  requirements  must  be 
satisfied 

Optimize  cost  and  minimize 
disruption 

Enterprise  focus 

Ensure  appraisal 
reliability 

Create  repeatable 
processes  i  standardize 

Desire  for  objectivity  increases 

Make  results  predictable 
and  differences  explainable 

Use  of  external  resources 
increases 

Results  independent  of 
team  composition 

Slide  adapted  and  updated  from  presentations  by  Mr.  Byrnes  while  managing  the  appraisal  project  at  the  SEI. 
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Idressing  Common  Risks 


Risk 

Factors 

High  Maturity 
Counter  Points 

Insufficient  senior 
mgt.  commitment 

Caused  by  turnover  or  mergers 

Based  on  disillusionment  with  results 

Resulting  from  shifting  investment  priorities 

Management  changes 
generally  dond  retopothe 
process  or  the 
improvement  activities. 

Middle  mgt. 
resistance 

Overriding  pressure  for  project  performance; 
Incentives  on  delivery,  not  quality 

Doubt  about  seriousness  of  senior  leadership 

Always  a  factor. 

Customer  drivers  impact 
perspective. 

Inappropriate 

goals 

Level  5  in  1  year 

75  business  units  to  be  assessed  by  year  end 

Goals  not  based  on  level 
attainment 

Unrealistic 

expectations 

The  great  productivity  gap  related  to  managing 
change 

The  technology  adoption  curve  and  change 
management  awareness 

Lack  of  continuous  focus  on  process  improvement 

Data  driven. 

Knowledge  of  what  can  be 
achieved. 

Customer  focused. 

Crash 

implementations 

No  plans  or  long-term  perspective,  and  lack  of 
following  through  on  improvement  efforts 

Termination  of  activities  before  they  are 
institutionalized 

Lots  of  efforts  at  any  one 
time. 

Not  one  nmegaoeffort. 
Several  methods  in  tool  kit. 

Slide  from  Paul  Byrnes62"'^  ISD  Customer  Conference  presentation 
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Message:  Appraisals  as  Risk  Management 


Spend  extra  time  up  front  defining  the  organization  scope. 


❖  Take  an  integrated  approach  to  process  deployment. 

Target  a  model  scope  that  makes  sense  for  your  current 
state,  business  goals,  and  business  environment. 

Conduct  informal,  but  robust,  interim  appraisals  (Class  C, 
Class  B)  as  a  risk  reduction  technique. 

Frankly,  these  apply  to  all  appraisals,  high  maturity  units 
are  just  better  at  it. . .. 


These  lessons  are  paraphrased  from  one  of  ISD®  CMMI  customers,  as  reported  in  2003  in  a  public  forum 
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Some  Example  High  Maturity  Teams 


This  was 
the  roldestc 
appraisal 

- ► 


This  was 
the  fiatesto 
appraisal 

- ► 


Team  Size 

Days  on  site 
for  A 

Team  Comp.: 
External  - 
External  to  OU  - 
Internal  to  OU 

Effort  hours 
/Team  Member 

10 

15 

4T  OT  6 

134 

9 

10 

1  T  2  T  6 

93 

8 

10 

2  1  (1  and  1)T 

4 

96 

8 

10 

1  1  (2  and  2)  T 

3 

95 

8 

10 

2T  4T  2 

86 

t 


Notice  the 
trend! 


Is  there  are  trend?? 
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Lessons  Implemented!  Tailoring 


Some  key  SCAMPI  HM  tailoring  and  variations  from  the  standard 
process  commonly  used  in  the  past  and  for  low  maturity  events 

^  organization  preparation  starts  much  sooner 

■  more  time  allocated  to  the  entire  event  (if  attempting  full  coverage  and 
ratings  and  multi-discipline  events) 

■  more  preparation  time  allocated  to  designing  appropriate  interview 
sessions  (size,  scope,  type,  etc.) 

<$>  team  selection  and  composition  even  more  critical  V  high  maturity 
experience,  SPC  skills,  inside/outside  unit,  specialized  training 

<$>  longer,  integrated  organization  in-brief  needed  T  discussion  of 
goals,  models/baselines,  subprocesses  required 

need  for  automated  tools  increased  T  expansion  in  data  elements 
required  increases  need  for  different  approaches  to  recording  data 

Slide  adapted  from  pdb  SEPG  2001  presentation 
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Project  Selection  Challenges 


Organization  Coverage:  large  units  have  a  real  challenge  of 
showing  institutionalization  across  the  entity  when  only  reviewing  a 
small  set  of  projects  in  a  Class  A  T  how  many  instances  is  enough? 

^  Model  Coverage:  projects  with  institutionalized  practices  which 
reflect  model  requirements:  In  high  maturity  events,  the  need  to 
bring  in  additional  data  from  “non-focus” projects  increases. 

<$>  Life  Cycle  Coverage  :  This  effects  all  appraisals,  but  is  exacerbated 
in  level  4-5  events  due  to  natural  life  cycle  implementation 
durations  for  these  kind  of  measurement  intensive  processes. 

<$>  Functional  Coverage:  no  different  issues  than  in  a  typical  appraisal 
T  but  there  may  be  more  groups  that  need  to  be  covered. 
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Objective  Evidence  Challenges 


High  maturity  processes  demand  more  instantiations 
than  just  a  fbne  direct,  one  indirectoapproach. 

Example:  in  OID,  seeing  one  example  of  a  systems 
engineering  tool  being  deployed  is  woefully  incomplete 
forjudging  organization  institutionalization 

■  What  about  software? 

■  What  about  a  major  process  change? 

■  What  about  supplier  management? 

■  What  about  large  programs  that  maintain  their  own  baselines? 

■  What  about  IR&D  and  CR&D  projects? 
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Lessons  Implemented  -  Evidence 


Organize  objective  evidence  in  a  user-friendly  manner 

■  Must  provide  guidance  for  interpreting  objective  evidence 

■  Store  evidence  electronically  T  Use  automated  tooling. 

■  Review  the  evidence  for  consistency  before  the  event 


Develop  fihreadsoto  follow  high  maturity  concepts  in  a 
more  natural  and  flowing  manner  i  present  evidence  by 
“topic”  rather  than  CMMI  practice  buckets 


Use  interim  (C  and  B)  appraisals  to  incrementally  fbuildo 
the  appraisal  database  T  HM  events  are  typicaiiy  not  just 
a  big  bang  singie  event 
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Lessons  Implemented  -  Conduct 


<$>  Ensure  most  (all??)  team  members  get  insight  into  the  high  maturity 
practices  being  implemented 

■  Facilitates  the  final  consolidation  process 

■  Leverage  fbverlapsoand  ftlependenciesoin  the  model  (and  threads)  to  assign 
mini-teams 

■  Mini-teams  usually  have  nnside-outsideomembership  to  maximize  objectivity 
while  benefiting  from  nnsideroknowledge 


<$>  High  Maturity  events  require  different,  additional  interview  participants 

■  Example:  for  OID,  Internal  Research  and  Development  (IR&D)  projects 


Use  parallel  interview  sessions  for  some  self-contained  (e.g.,  SAM)  T 
maximize  time  for  whole  team  on  HM  sessions  and  tasks. 


Perform  parallel  splits  for  topics  that  are  generally  or  easy  to  parse  between  org 
an  projects  (e.g.,  OPF) 
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HM  Appraisal  Considerations 


Appraisal  practices 
(examples) 

Implementation 

issues/risks/recommendations 

Appraisal 

considerations 

Plan  the  Process  (GP  2.2) 

Organizations  often  donQ  know  how  much 
data  is  needed  relative  to  prior  events  when 
increasing  model  and  discipline  scope. 

Must  engage  outside  Lead  sooner  in 
internal  planning  stages. 

Sampling  strategies 

Identify  and  Involve 
Stakeholders  (GP  2.7) 

Very  broad  set  of  stakeholders.  Easy  to  miss 
key  people.  May  involve  groups  not 
previously  part  of  low  maturity  appraisals. 

When  ftiewO  groups  involved,  they 
exhibit  Plow  appraisal  maturityO 
despite  organization  overall  process 
capability. 

Establish  a  Defined 

Process  (GP  3.1) 

Organizations  often  focus  on  procedures 
within  processes,  rather  than  with  interfaces, 
coordination,  synergy,  and  integration 
across. 

Look  for  threads.  Sets  of  documents 
that  describe  connections  across 
process  elements. 

Review  Status  with  Higher 
Level  Management  (GP 
2.10) 

Many  issues  and  decisions  can  be  driven 
down  to  lower  levels  i  delegate 
responsibility. 

Manage  the  effort  like  a  project. 
Decompose  the  problem.  Track 
metrics.  Set  norms  up  front.  Do 
training  even  if  they  already  had  it. 

Manage  Configurations 
(GP  2.6) 

Data  across  company  likely  to  be  in 
multiple  repositories.  Significant  IT, 
security 

Need  for  good  CM  to  manage 
incremental  appraisal  database  build 
up  and  reuse  over  several  events. 
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Common  Pitfalls  in  HM  Appraisals 


rProcess  improvement  team  centric  PIIDse  .6 

fSince  this  is  a  L5  appraisal,  it  has  to  take  4  weekse  6 

fSince  I  am  the  same  Lead  Appraiser  that  appraised 
you  last  time,  this  HM  event  will  be  easye  .6 

fWe  have  been  doing  this  forever,  let®  just  hire  the 
Lead  Appraiser  two  months  before  the  A.6 

We  hired  a  great  SPC  consultant  to  help  us,  let®  not 
worry  about  interacting  with  our  Lead  regarding  our 
interpretationse  .6 

We  were  HM  last  time,  why  do  we  need  to  be 
concerned  with  SEI  nowe  ?6 
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3araphrases  some  terminology  discussed  in  the  SCAMPI  Lead  Appraiser 
^/ledge  (BOK)  and  examples  generated  in  the  BOK  workshop  last  winter. 


Appraisal  Project  Management 

Planning  phase  is  longer  than  a  f1ypical6L2-3  appraisal 

Ensure  LA  counters  pre-disposition  to  spend  less  effort  in  diligence 
on  lower  maturity  PAs 

^  Align  all  applicable  goals  and  objectives 

■  Organization®  business  objectives,  PI  objectives,  Quality  and  Process 
Performance  Objectives  AND  the  appraisal  objectives 


Use  of  appraisal  historical  data  for  planning 


More  sophisticated  sampling  approaches 
LA  fimodelsohigh  maturity  behavior 
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Increased  Skills  Needed 


^  Integration,  Articulation  and  Expression  of  Information 

■  Increased  need  for  specialized  communication  skills 

■  Ability  to  describe  behavior  with  examples/scenarios/stories  i  thread 
based  appraisal  rather  than  rpractice  basedoappraisal 

■  Ability  to  express  infrastructure  necessary  to  successfully  implement 
L4-5  [e.g.,  IPM  tailoring  to  L4  QPM  metrics  flailoring^ 

Understanding  and  Adapting  to  Organizational  Context 

■  Understanding  Business  Goals  and  Concerns,  Understanding 
Organization  structure,  context,  environment,  and  culture,  and 
activities  deployed  to  resolve  problems 

^  Examining  High  Maturity  Organizational  Behavior 

■  Knowing  what  to  look  for  and  what  to  ask  about  (Both  org  and  project) 

■  Understanding  model  interpretations  (not  just  literal  words  of  model, 
but  intent) 
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Increased  Skills  Needed  i  2 


^  Understanding  an  array  of  quantitative  and  statistical  management 
metrics/techniques  that  may  be  applicable  depending  on  the  context 

■  Ability  to  differentiate  statistical  from  quantitative  methods 

■  Ability  to  accept  appropriate  quantitative  methods  as  reasonable  L4-5  behavior 

■  What  is  the  answer  to  fhow  much  is  enoughoHM  application  in  different  settings 

<$>  Greater  emphasis  on  need  to  understand  change  management  and 
technology  transition  methods 

<$>  Ability  to  fintegrateorather  than  de-compose  [holistic  perspective} 

<$>  Ability  to  explain,  and  reach  agreement  on,  HM  concepts  with  sponsors, 
participants,  and  team  members 

^  HM  appraisals  tend  to  shift  burden  on  LA  in  what/how  to  communicate  to 
stakeholders  (due  to  increased  skills  of  sponsor/team  members) 


Copyright  ©  2007  ISO,  Inc. 


Lessons  Learned  Conducting  High  Maturity  SCAMPIs 


17 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


Model  Interpretation  Issues  i  1 


^  What  is  enough  application  of  a  quantitative  technique? 
^  Characterization  and  rating  T  CL  vs.  ML 
^  Interrelationships  and  iterative  nature  with  CL-ML4/5 
L4&5  as  evolution  of  L2&L3;  not  distinct/separate 


^  Subpractices  and  informative  materiais  have  “heavier 
weight”  at  ML4/5?  [See  aiso  severai  recent  SEI 
briefings  corroborating  this] 
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Model  Interpretation  Issues  i  2 

How  much  is  enough  implementation  evidence,  how 
much  appropriate  SPC/quantitative  analysis,  etc. 

ftiust  making  itoversus  continuing  to  evolve,  etc. 

Recognize  when  appropriate  tools,  techniques,  etc  are 
being  applied  (viable  vs.  rgood^ 

Life  after  Level  5  T  show  things  continuing/evolving  on 
reappraisal;  how  much  improvement  do  we  need  to 
see? 
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Common  Pitfalls  Implementing  HM  Practices 


<$>  fDk,  it  took  us  1 2  months  to  do  L3,  wedi  be  able  to  L4  is 
6  monthse  .6 

^  fWe  have  one  good  example  of  SPG  in  engineering, 
why  would  you  want  to  see  more  e  .6 

fWe  do  one  control  chart  great,  we  just  forgot  about  all 
our  other  metricse  6 

^  fWe  do  six  sigma,  therefore  we  are  L5e  .6 

fCorporate  has  two  process  performance  models  i  they 
dond  relate  to  what  we  do  in  this  unit,  but  OPP  is  oke  .6 

fWe  do  causal  analysis,  we  must  be  L5  e  .6 

fWe  have  lots  of  pretty  charts,  what  else  would  we 
neede  6 
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Some  Key  High  Maturity  Take  Aways 


^  Management  is  heavily  embedded  in  the  process. 

^  High  maturity  organizations  can  manage/sustain 

performance  in  spite  of  routine  organizational  fshocks.o 

^  Direct  customer/user  involvement  in  the  improvement 
process  is  high. 

^  No  single  hmethodoor  nmodeloused  T  a  tool  kit  is  used. 

Most  are  not  doing  the  practices  because  they  want 
Level  4-5. 
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Different  Behaviors  in  HM  Orgs  (Really!) 


The  organization  keeps  an  eye  on  the  outside  world  for 
innovations. 

High  fpeople  to  peopleoguidance  provided.  Much  more  ncoaching.6 

❖  Current  and  desired  capability  of  processes  is  understood. 
Variations  across  tailoring  parameters  is  known  and  factored. 

Work  is  aligned  with  business  objectives  and  customer  needs. 

Many  nadditionaloroles  are  actively  involved. 

flntegrated  teamsoreview  and  analyze  data,  and  make 
improvements. 
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Key  HM  Organizational/Appraisal  Challenges 

Organizational 

■  Too  many  models.  Too  many  methods.  Multi-model  appraisals. 

■  Management  drivers  for  reduced  costs. 

■  Increasing  efficiency  of  both  internal  improvement  and  external 
appraisal  efforts. 

■  Customer  ndisconnectsobetween  fievel  achievementoand  rproject 
performance. 6 

<$>  Appraisal 

■  May  be  hard  for  organizational  participants  to  ndescribeothings  to 
external  team  members. 

■  Thread  based  appraisal  vs.  practice  based  appraisal 

■  Data  element  needs  increase  substantially. 

■  Some  SCAMPI  rules  can  actually  get  in  the  way 
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Background 


■  As  a  set,  the  SCAMPI  methods  provide  a  powerful  set 
of  tools  to  use  in  CMMI  adoption 

■  However,  there  are  some  situations  in  which  these 
three  methods  are  not  appropriate,  or  are  not  cost- 
effective 

■  This  presentation  will  discuss  the  features  and 
limitations  of  the  three  methods,  and  alternatives  that 
should  be  considered 
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Characteristics  of  CMMI  Appraisai  Ciasses 


The  ARC  (Appraisal  Requirements  for  CMMI)  defines  appraisal  classes 

■  A  guide  to  inventors  of  appraisal  methods,  and  their  customers 

Key  differentiating  attributes  for  appraisal  classes  include 

•  the  degree  of  confidence  in  the  appraisal  outcomes 

•  the  generation  of  ratings 

•  appraisal  cost  and  duration 

Appraisal  Requirements  for  CMMI,  Version  1.1,  CMU/SEI-2001-TR-034 


Characteristics 

Class  A 

Class  B 

Class  C 

Amount  of  Objective  Evi¬ 
dence  Gathered  (relative) 

Eijzh 

Mediimi 

Low 

Ratings  Generated 

No 

No 

Resource  Needs  [relative) 

Hi^Ji 

Mediimi 

Low 

Team  Size  (relative) 

Lai’^e 

Mediimi 

Small 

Appraisal  Team  Leader 
Requirements 

Lead  appraisei’ 

Lead  appiai&ei'  or 
per-son  traii’.ed 
and  experienced 

Persoi:  tiaiued 
and  experienced 

References:  “A  Quantitative  Comparison  of  SCAMPI-A  SCAMPI-B 

SCAMPI  A,  B,  and  C,  ”  R.  Hefner  and  D. 

Luttrell,  CMMI  Technology  Conference  and 
User  Group,  2005 

Hefner,  "Lower  Cost,  More  Effective  Alternatives  to  SCAM  Pis",  20Qj 


SCAMPI-C 


GRUMMAN 
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A  Variety  of  Appraisals 


All  Appraisals 
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What's  Important  About  ARC  Compliance? 


The  appraisal  principles  for  the  CMMI  Product  Suite  are  similar  to  those  for 
appraisals  using  the  Capability  Maturity  Model  for  Software  and  Systems 
Engineering  Capability  Model: 

■  Start  with  an  appraisal  reference  model. 

■  Use  a  formalized  appraisal  process. 

■  Involve  senior  management  as  the  appraisal  sponsor. 

■  Focus  the  appraisal  on  the  sponsor’s  business  objectives. 

■  Observe  strict  confidentiality  and  non-attribution  of  data. 

■  Approach  the  appraisal  collaboratively. 

■  Focus  on  follow-on  activities  and  decision-making  based  upon  the 
appraisal  results. 

-ARC,  v1.2 


■  In  what  situations  would  these  principles  not  be  appropriate? 

■  Sponsor  desire  for  an  informal  appraisal  process 

■  Non-attribution  not  critical 

■  Inability/no  desire  to  work  collaboratively 
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What's  Important  About  SCAMPI-A 
Compliance? 


The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  is  designed 
to  provide  benchmark-quality  ratings  relative  to  Capability  Maturity  Model  Integration 
(CMMI)  models. 

SCAMPI  A  enables  a  sponsor  to 

■  gain  insight  into  an  organization’s  capability  by  identifying  the  strengths  and 
weaknesses  of  its  current  processes 

■  relate  these  strengths  and  weaknesses  to  the  CMMI  reference  model(s) 

■  prioritize  improvement  plans 

■  focus  on  improvements  (correct  weaknesses  that  generate  risks)  that  are  most 
beneficial 

■  to  the  organization  given  its  current  level  of  organizational  maturity  or  process 
capabilities 

■  derive  capability  level  ratings  as  well  as  a  maturity  level  rating 

■  identify  development/acquisition  risks  relative  to  capability/maturity  determinations 
_ -SCAMPI A,  v1.2 

■  SCAMPI-A  appraisals  were  designed  to: 

■  Be  accurate  (collaboration  of  multiple  sources  -  direct,  indirect,  written/face-to-face 
affirmations,  trained  team,  authorized  team  leader) 

■  Achieve  organizational  buy-in  (collaborative  approach,  construction  of  PIIDs, 
interviews,  draft  findings) 
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How  Do  SCAM  PI- B  and  -C  Relate? 


These  methods  can  form  building  blocks  for  a  progression  of  appraisals  -  for 
example,  starting  with  a  SCAMPI  C  reviewing  the  process  descriptions,  then  a 
SCAMPI  B  investigating  their  deployment  to  projects,  finally  leading  to  a  formal 
benchmarking  event  focused  on  institutionalization  of  the  practices  across  the 
organization. 

“  Handbook  for  Conducting  Standard  CMMI  Appraisal  Method  for  Process 

Improvement  (SCAMPI)  B  and  C  Appraisals,  Version  1. 1 


■  But  all  SCAMPI  appraisals  share  the  same  basic  methods 
(interviews,  evidence  review,  team  qualifications)  and  reflect 
similar  objectives  (accuracy,  buy-in) 

■  The  typical  SCAMPI  C/B/A  sequence  works  well  for  an 
organization  starting  a  process  improvement  effort,  i.e.,  no 
defined  processes 

■  May  not  work  as  well  for  an  organization  that  has  existing 
processes,  and  whose  main  issue  is  project  adoption 

NORTHHOR  GRUMMAN 
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Adopting  the  CMMI 


■  Key  enablers 

■  Willingness  to  learn  unfamiliar  practices 

■  Desire  to  extract  value  rather  than  “check  the  box” 

■  Ability  to  interpret  the  CMMI  in  your  context 

■  Access  to  experts 
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Effective  Use  of  Audits  and  Appraisais 


■  Process  and  product  audits  provide  tangibie,  objective 
measures  of  adoption/sustainment 

■  Policies,  processes,  and  standards  must  reflect  the  desired 
behaviors 

■  Appraisals  evaluate  the  effectiveness  of  the  audit  program 

■  Standardized  tools,  approaches,  and  methods 

■  Consistency  of  appraisers  -  if  they  understand  the  way  we  are 
structured  and  operate,  there  is  less  time  required  to  understand 
what  we  are  doing. 

■  Pre-appraisal  activities  to  prepare  projects  for  the  appraisal 
process 

■  The  frequency  of  audits  and  appraisals,  and  the  sampling,  must 
reflect  the  progress  of  the  cultural  change 

■  As  the  culture  begins  the  change,  more  frequent  and  more  in-depth 
audits/appraisals  are  required 

■  Later,  the  amount  of  audits/appraisal  may  decrease,  jf  the  culture 
has  truly  changed 


“Sustaining  CMMi  Compliance ,  ”  R.  Hefner,  CMMI  Technology  Conference  and  User  Group,  2006 
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Where  Could  We  Save  Money? 


■  Could  we  ignore/relax  some  of  the 
ARC  requirements? 

■  Use  an  undocumented  method 

■  Use  an  untrained  team 

■  Less  preparation  of  participants 

■  Less  invoivement  of  participants 

■  Less  corroboration  of  evidence 


■  Could  we  use  different  approaches  than  SCAMPI  uses? 

■  Assist  projects  in  evidence  gathering 

■  Don’t  require  consensus  among  appraisers 

■  Use  a  different  rating  scheme  (or  no  ratings) 

■  Use  different  objectives  than  practice  compiiance  (efficiency, 
effectiveness,  consistency,  understanding/awareness,  etc.) 
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Impacts 


“A  Quantitative  Comparison  of  SCAMPi  A,  B,  and  C,  ”  R.  Hefner  and  D.  Luttreii, 

CMMi  Technoiogy  Conference  and  User  Group,  2005  NOnmR€>R  CRWtM/kM 

Hefner,  "Lower  Cost,  More  Effective  Alternatives  to  SCAM  Pis",  20Qj 
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Minimum  Team  Size 


Cost  is  composed  of: 

■  Team  costs  -  goes  up  with  team  members 

■  Organizational  costs  (interview,  presentations) 
-  iargeiy  fixed  regardiess  of  size 


■  Accuracy  goes  up  with  as  team  size  increases 


Buy-in  is  driven  by  the  confidence  the 
organization’s  members  has  in  the  appraisal 
process  and  appraisal  team 

■  Larger  teams  can  increase  the  iikeiihood  that  a 
respected  person  is  on  the  team 
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Team  Accuracy  vs.  Team  Size 


"  Same,  assuming  90%  leader 
accuracy 


As  team  size  goes  up,  team 
accuracy  rapidly  increases 
(assuming  the  right  answer  is 
obvious  once  presented) 

Teams  of  greater  than  4  provide 
littie  increase  in  accuracy 


If  the  team  leader  is  90% 
accurate,  additionai  team 
members  add  little  accuracy 

Adding  team  members  does  give 
a  chance  for  them  to  learn 


Appraiser  accuracy,  not  team  size,  is  criticai 

GRUMMAN 
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Sources  of  Objective  Evidence 


Evidence  review  takes  1-2  times  the  length  of 
interviews 

■  If  evidence  is  not  reviewed,  easy  to  answer 
“correctly”  in  the  interviews 

■  If  interviews  are  not  conducted,  evidence  may  be 
faked  (not  really  in  use)  -  normally  easy  to  spot 


Accuracy  increases  significantly  with  evidence 
review 

Validation  takes  littie  time  and  often  increases 
accuracy  20-30% 


Buy-in  is  greatiy  increased  by  validation 

■  Nothing  decreases  buy-in  faster  than  a  “weakness” 
that  everyone  knows  is  wrong 
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The  Workshop  Concept 


■  Determine  effective  ways  to  perform  and/or  document  practices 

■  Raise  awareness  of  project  personnei,  buiid  buy-in 

■  Process: 

1 .  Train  projects  on  CMMi  terminology  and  structure  (1  -3  day) 

2.  Projects  compiete  PliDs  mapping  of  their  existing  evidence, 
self-assess  practice  and  evidence  gaps 

3.  A  CMMi  expert  waiks  a  group  of  projects  through  the  modei.  For 
each  practice,  the  expert: 

■  Describes  the  practice  and  typicai  evidence 

■  Reviews  each  project’s  evidence  for  acceptabiiity 

■  Identifies  practice  gaps  and  discusses  possibie  soiutions 

■  Identifies  documentation  gaps  and  possibie  soiutions 

NOHTHftOR  GmmMJUH 
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Summary 


■  As  a  set,  the  SCAMPI  methods  provide  a  powerful  set 
of  tools  to  use  in  CMMI  adoption 

■  However,  there  are  some  situations  in  which  these 
three  methods  are  not  appropriate,  or  are  not  cost- 
effective 

■  Improvement  professionals  should  consider  the  full 
range  of  options  available  to  them,  and  select  the  tools 
and  methods  best  suited  to  the  needs  of  the  sponsor 
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Background 


■  The  hardest  part  of  implementing  CMMI-based 
improvements  is  getting  projects  to  understand  and 
perform  the  practices 

■  Workshops  can  be  an  effective  mechanism  for: 

■  Raising  awareness  and  buy-in 

■  Developing  a  deeper  understanding  of  the  practices 

■  Ensuring  they  are  properly  implemented  by  the  project 
personnel 

■  This  presentation  will  explain  how  to  plan  and  conduct 
CMMI  workshops,  based  on  the  proven  methods  used 
by  Northrop  Grumman  in  achieving  Level  5  across  13 
organizations 
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Topics 


■  When  the  typical  SCAMPI  C/B/A  sequence  doesn’t 

■  The  workshop  concept 

■  How  to  scope  and  plan  the  workshop 

■  Choosing  workshop  participants 

■  Identifying  the  “right”  evidence 

■  Additional  opportunities 

■  Dealing  with  resistance  and  lack  of  buy-in 

■  Workshop  follow-up 

■  Sustaining  senior  management  support 

■  Lessons  Learned 


work 
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Characteristics  of  CMMI  Appraisai  Ciasses 


The  ARC  (Appraisal  Requirements  for  CMMI)  defines  appraisal  classes 

■  A  guide  to  inventors  of  appraisal  methods,  and  their  customers 

Key  differentiating  attributes  for  appraisal  classes  include 

•  the  degree  of  confidence  in  the  appraisal  outcomes 

•  the  generation  of  ratings 

•  appraisal  cost  and  duration 

Appraisal  Requirements  for  CMMI,  Version  1.1,  CMU/SEI-2001-TR-034 


Characteristics 

Class  A 

Class  B 

Class  C 

Amount  of  Objective  Evi¬ 
dence  Gathered  (relative) 

Eijzh 

Mediimi 

Low 

Ratings  Generated 

No 

No 

Resource  Needs  [relative) 

Hi^Ji 

Mediimi 

Low 

Team  Size  (relative) 

Lai’^e 

Mediimi 

Small 

Appraisal  Team  Leader 
Requirements 

Lead  appraisei’ 

Lead  appiai&ei'  or 
per-son  traii’.ed 
and  experienced 

Persoi:  tiaiued 
and  experienced 

SCAMPI-C 


References:  “A  Quantitative  Comparison  of  SCAMPI-A  SCAMPI-B 

SCAMPI  A,  B,  and  C,  ”  R.  Hefner  and  D. 

Luttrell,  CMMI  Technology  Conference  and 

User  Group,  2005  _ 
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When  the  Typical  SCAMPI  C/B/A  Sequence 
Doesn't  Work 


These  methods  can  form  building  blocks  for  a  progression  of  appraisals  -  for 
example,  starting  with  a  SCAMPI  C  reviewing  the  process  descriptions,  then  a 
SCAMPI  B  investigating  their  deployment  to  projects,  finally  leading  to  a  formal 
benchmarking  event  focused  on  institutionalization  of  the  practices  across  the 
organization. 

-  Handbook  for  Conducting  Standard  CMMI  Appraisal  Method  for  Process 

Improvement  (SCAMPI)  B  and  C  Appraisals,  Version  1.1 


■  The  typical  SCAMPI  C/B/A  sequence  works  well  for  an 
organization  starting  a  process  improvement  effort, 
i.e.,  no  defined  processes 

■  May  not  work  as  well  for  an  organization  that  has 
existing  processes,  and  whose  main  issue  is  project 
adoption 


NORTHROR  CRUMM/XN 

"Using  Workshops  to  Speed  CMMI  Adoption  and  Evidence  Gathering'^jJ^ddT^ 

Copyright  2005  Northrop  Grumman  Corporation 


Adopting  the  CMMI 


"  Key  enablers 

■  Willingness  to  learn  unfamiliar  practices 

■  Desire  to  extract  value  rather  than  “check  the  box” 

■  Ability  to  interpret  the  CMMI  in  your  context 

■  Access  to  experts 
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The  Workshop  Concept 


■  Determine  effective  ways  to  perform  and/or  document  practices 

■  Raise  awareness  of  project  personnei,  buiid  buy-in 

■  Process: 

1 .  Train  projects  on  CMMi  terminology  and  structure  (1  -3  day) 

2.  Projects  compiete  PliDs  mapping  of  their  existing  evidence, 
self-assess  practice  and  evidence  gaps 

3.  A  CMMi  expert  waiks  a  group  of  projects  through  the  modei.  For 
each  practice,  the  expert: 

■  Describes  the  practice  and  typicai  evidence 

■  Reviews  each  project’s  evidence  for  acceptabiiity 

■  Identifies  practice  gaps  and  discusses  possibie  soiutions 

■  Identifies  documentation  gaps  and  possibie  soiutions 
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How  To  Scope  And  Plan  The  Workshop 


■  Several  projects  can  participate  at  the  same  time 

■  Explain  once  to  many  projects,  build  off  each  other’s  questions 

■  Can  use  projects  who  are  performing  the  practice,  or  documenting 
properly  as  examples 

■  Peer  pressure 

■  Having  muitiple  projects  means: 

■  More  frequent  context  switching  by  the  CMMI  expert 

■  More  logistics 

■  Best  practices 

■  CMMI  expert  should  become  familiar 
with  each  project’s  context,  terminology 

■  One  process  area  per  session  with 
process  area  performers 

■  Front  screen  display  of  the  PI  IDs  table 

■  Each  project  uses  a  separate  computer 
for  their  PUDS,  evidence  display 
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Choosing  Workshop  Participants 


■  The  performer(s)  of  the  process  should  be  present 

■  Explain  implementation  and  evidence 

■  Explain  context  and  project  culture  (e.g.,  barriers) 

■  If  practice  is  not  currently  being  performed,  discuss  the  value  of  the 
practice,  and  possible  approaches  that  might  be  value-added 

■  If  practice  is  being  performed  but  not  documented,  discuss  possible 
documentation  approaches  that  fit  the  culture 
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Identifying  The  Right  Evidence 


■  Because  so  much  of  the  focus  is  on 
finding  direct  evidence  for  each  practice, 
it  is  easy  to  forget  that  the  objective  is 
improving  the  process 

■  Challenges 

■  Bring  Me  a  Rock 

■  “If  our  document  said _ ,  wouid  that  be  enough?” 

■  Documenting  for  the  appraisers,  not  the  project  personnei 

■  Remember:  the  purpose  of  plans  and  processes  is  to  provide 
guidance  to  the  project  personnei 

■  Appraisers  can  suggest  what  items  shouid  be  covered 

■  Adequacy  is  determined  by  whether  project  personnei  understand 
what  to  do 
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Additional  Opportunities 


■  Can  conduct  simultaneous  quality  assurance  process  audits 

■  Appraise  against  the  projects  defined  process  (which  probably 
includes  all  the  CMMI  practices) 

■  Educate  the  QA  staff  on  the  proper  approach  to  an  audit,  and  the 
terminology/meaning  of  the  CMMI  practices 

■  Can  look  for  other  process  Improvement  opportunities  beyond 
CMMI  compliance 

■  Consistency  across  the  organization 

■  Identification  of  best  practices 

■  Efficiency,  effectiveness 

■  Need  for  tools,  templates,  training 
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Dealing  With  Resistance  And  Lack  Of  Buy-in 


■  Workshops  offer  a  great  opportunity  to  gauge  project 
understanding  and  buy-in  to  the  improvement  effort 

■  Do  the  project  personnel  make  a  honest  effort  to  map  their 
evidence? 

■  Do  they  show  up  on  time  and  prepared? 

■  Do  they  appear  engaged  in  determining  soiutions? 

■  Are  they  iooking  to  improve  their  processes,  or  just  satisfy  the 
appraisers? 

■  What  factors  are  preventing  their  compiete  commitment  (time, 
knowledge,  management  encouragement,  etc.) 
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Workshop  Follow-up 


■  Each  workshop  results  in 

■  A  set  of  practice  gaps  and  proposed  approaches 
(start  doing  this) 

■  A  set  of  documentation  gaps  and  proposed  approaches 
(start  documenting  what  we  are  currently  doing  like  this) 

■  These  should  be  converted  into  a  set  of  actions  and  timelines 

■  When  will  the  evidence  exist,  so  we  can  re-assess? 

■  Tracking  against  this  timeiine  wiil  teli  you  when  you  wiil  be  ready 
for  another  workshop  and  eventualiy,  a  more  formal  appraisal 

■  A  second  group  session  is  sometimes  useful 

■  Isolated  gap  closures  can  be  handled  one-on-one 
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Sustaining  Senior  Management  Support 


■  Senior  management  should  be  kept  appraised  of  progress  and 
barriers  to  achieving  their  goals 

■  Number  of  current  gaps  and  rate  of  closure 

■  Common  gap 

■  Opportunities 

■  Resistance 
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areas 

beyond  CMMI  compliance 


Total  Number:  217 
Missing  ECDs:  0  0% 

Missing  Schedules:  0  0% 

Total  Closed:  180  83% 
Total  Late  to  Schedule:  1  0% 
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Lessons  Learned 


■  The  hardest  part  of  implementing  CMMI-based  improvements  is 
getting  projects  to  understand  and  perform  the  practices 

■  Workshops  can  be  an  effective  mechanism  for: 

■  Raising  awareness  and  buy-in 

■  Developing  a  deeper  understanding  of  the  practices 

■  Ensuring  they  are  properly  implennented  by  the  project  personnel 

■  Engaging  with  the  projects,  and  understand  their  barriers  to 
improvement,  is  the  true  spirit  of  process  improvement 
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"All  models  are  wrong, 
but  some  are  useful.” 
George  Box 

(Quality  and  Statistics  Engineer) 


A  A  CMMI  model  is  not  a 
process. 


A  A  CMMI  model  describes  the 
characteristics  of  effective 
processes. 
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CMMI 


Process  Area 


1.  Function 


3.  Implementation 


Benefits 


The  Process  Areas  are 

used  as  building  blocks  to  construct  a 

foundation  for  improving  process 

performance. 


The  practices  in  the  PAs  provide 
organizations  a  set  of  proven 
management  tools  that  are  non- 
prescriptive  (never  a  set  of 
implementation  practices). 

Each  organization  should  determine 
howto  implement  these  practices  within 
their  organizations  always  from  a 
pragmatic,  “what  makes  sense” 
perspective. 
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These  benefits  of  formal 
process  improvement  activities 
and  SCAMPI  Class  C  appraisals 
are  more  applicable  to  small  and 
medium  organizations  than 
larger  corporations. 


The  small  to  medium  organizations 
often  function  as  suppliers  of 
specialized  technical  services  or 
products. 


These  benefits  of  SCAMPI  C 
appraisals  are  from  a  sample  of 
small  to  medium  organizations  that 
continued  their  process 
improvement  journeys  by 
conducting  a  SCAMPI  B  and 
planning  a  Class  A  . 
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SCAMPI  strategy 


SCAMPI  Family  of 
Appraisals 


SCAMPI  A 


A  key  activity  in  obtaining  a 
SCAMPI  benchmark 
is  applying  the 
risk  management 

functions  of  SCAMPI  SCAMPI  B 

Class  C  and  B  appraisals 
before  scheduling  a 
Class  A  benchmark. 
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Success  Factor  Benefits  of  SCAMPI  C 
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Small  &  medium  organizations  are  not 
“miniatures”  of  large  corporations! 


Smaller  organizations  provide  a  conducive 
environment  to  implement  CMMI  practices 
due  to: 

1 .  simplicity  of  organizational  structure 

2.  efficient  communications 

3.  staff  receptiveness  of  new  ideas 

4.  depth  of  awareness  of  the  processes 

5.  easier  to  minimize  variance  in 
performing  key  processes 
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Evaluated  4  competing  project  evaluation  models  .  . 

-  ISO  (base  case) 

-  OPM3  (published  by  Project  Management  Institute  -  PMI) 

-  CMMI  ver  1.2 

-  Kersner^  (proprietary  published  model) 


■  .  .  .  Against  5  criteria: 

-  Credibility  and  wide-use  in  industry 

-  Identifies  crisp  and  actionable  items 

-  Holistic  and  systematic 

-  Cost  to  evaluate  and  maintain 

-  Proven  correlation  to  business  improvement 


^Using  the  Project  Management  Maturity  Model,  2"**  edition,  2005,  Harold  Kerzner,  PhD,  ISBN  0-471-69161-5 


i.  HHC  complimentary 


^  ^  nil"  period  has  ended. 

■»  Complete 

to 

and  Expanded  Features 

Alternative  Analysis 

H  Honeywell  FM&T 

H  Kansas  City  Plant 

m 

Honeywell 


1  - “ - 

— — uoncepts 

Goals  — — 

WF 

Base 

Case: 

ISO  9000 

CMMI 

Kei'zner 

OPM3 
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widely  in  commercial  industry 

2.  The  model  identifies  crisp  and  actionable 
improvements 

3.  The  model  drives  a  holistic  and  systematic 
approach  to  driving  enterprise  improvements 

4.  Cost  to  evaluate/implement/sustain 

5.  The  model  has  a  proven/demonstrated 
correlation  to  improved  enterprise  results. 
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Appraisal  Scope  using 
Sntinuous  Representation 


Risk  Management 
was  important  to 
the  NNSA 
customer  and  had 
been  a  focus  of 
the  organization 
for  the  previous 
years. 

The  Continuous 
Representation 
allowed  the 
flexibility  to 
include  RSKM  in 
the  appraisal. 


Category 

Process  Areas 

Process 

Management 

r^roC333  f  33LJ3 

Dr^r];jj2;rjj:jD;j3j  PrD3333  DBT\n\i\on 

Os:<^c\n\-Z'c.mom\\  '^rcnn^ng 

PrD3333  Penomjrj/jce 

Orcjii/JEHiJD/JriJ  J/j/JoyHiJo/j  si/jd  DspJoyi/Jsm 

Project 

Management 

Project  Planning 

Project  Monitoring  and  Controi 

Supplier  Agreement  Management 

Jrjiegmied  Projeci 

Risk  Management 

QLJ'imj'hnJVs  Projaci  Managaniani 

Engineering 

Requirements  Management 

p3!:jLJjr3/j'J3m3  DavalopsnafYL 

Tsah/jJCiiJ  SoJuilo/j 

Ps'oiluaL  Iriiagraaori 

yanncavon 

yiiJldsiilo/j 

Support 

Configuration  Management 

Process  and  Product  Quaiity  Assurance 

Measurement  and  Analysis 

C5iiJ35iJ  y-\j'j'iJy3J3  and  P93DJLJdorj 

D33J3JDJ'J  y-\/J'lJy3J3  and  p33DJLJdo/J 
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II 


Initiation 


Phase  Gates 


V 


Maintenance 


Waiting  Approval 


Request 

- 

Gathering 

Cancel  or 

- 

Requirements 

► - 

Complete 

A 


Design 


Construction 


<7 


Close-Out 


Released  for 

_ f’ 

Design 

- [■ 

Close-Out  Process 


Released  for 

_ t'. 

Construction 

- ^ 

Design 


i>  Construction 


{>  Closed 


Planning  & 

Release  for  Ping  & 

- 
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Project  Ping 

Development 
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Challenge 


Mapping 
Construction 
Language  to 
CMMI 


Configuration  Management 


Specific  Goal  and 
Practices 

Typical  Work  Product 

Process/Tool  that  satisfies  SP 

Link  to 

Process/Tool 

SG  1 

Establish  Baselines 

SP  1.1 

Identify  Configuration 
Items 

Scope 

How  to  Control  Authorized  Proiects 

04.01.01.04.37 

Schedule 

Budget 

SP  1.2 

Need  system  Description 

File  System 

Project  Records 

04.01.01.04.35 

Command  Media 

Facilities  Reference  Manuals 

04.01.01.04.21 

Project  Database 

Process  Maps 

QA  Manual 

SP  1.3 

Create  or  Release 
Baselines 

Project  Charter 

Database 

SOW 

EVMS  Work/Budget  Authorization 

04.01.01.04.37 

Design  Criteria 

How  to  Request  Project  Authorizations 

04.01.01.04.08 

Drawings  &  Specs 

Proiect  Layouts 

04.01.01.04.22 

PEP 

How  to  Prepare  Line  Item  Documents 

04.01.01.04.04 

Authorization  Documents 

How  to  Prepare  GPP  Documents 

04.01.01.04.45 

SG2 

Track  and  Control  Changes 

SP2.1 

Track  Change 

Requests 

emails 

How  to  Perform  Proiect  Change  Control 

01.04.04.00.18 

Q-Reviews 

EVMS  Change  Incorporation 

04.01.01.04.37 

Authorization  Mods  &  BCP 

How  to  Control  Authorized  Proiects 

04.01.01.04.37 

Project  Database 

SP2.2 

Control  Configuration 
Items 

Proiect  Files 

How  to  Close-out  Facilities  Projects 

04.01.01.04.39 

How  to  Disposition  records 

01.06.05.00.04 

SG3 

Establish  Integrity 

SP3.1 

Establish 

Configuration 
Management  Records 

Project  Database 

Change  Orders 

EVMS  Subcontract  Management 

04.01.01.04.37 

Submittals 

Construction  Management  Manual 

SP3.2 

Perform  Configuration 
Audits 

Audits 

Project  Records 

04.01.01.04.35 

Q- Reviews 

BOI 

How  to  Disposition  records 

01.06.05.00.04 

Project  Closing  Review 

How  to  Close-out  Facilities  Projects _ 

04.01.01.04.39 
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Jeanie  Kitson,  President,  KAMO  Consultancy,  LLC  (Appraisal  Team  Lead) 

Dave  Kitson,  Vice  President,  KAMO  Consultancy,  LLC 

Paul  Kimmerly,  SEPG  Lead,  US  Marine  Corps  Technology  Services 

Organization,  Kansas  City 

Valerie  Tourangeau,  Director  of  Corp  IT  Global  Quality  Programs,  Honeywell 

Steve  Stafford,  Construction  Oversight  Manager,  FES,  Honeywell  Kansas  City 

Plant 

Craig  Nordeen,  Cost  Engineer,  FES,  Honeywell  Kansas  City  Plant 
Randy  Hamilton,  Project  Director,  FM&T,  Honeywell  Kansas  City  Plant 
Larry  Stotts,  Project  Engineer,  FES,  Honeywell  Kansas  City  Plant 
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Appraisal  Interviewees  and 
Document  References 


1  Sponsor 
5  Project  Managers 
1  Project  Director 
1  Team  Manager 
1  Title  III  Engineer 

1  Construction  Manager 

2  Planners 

2  Cost  Engineers 
1  Architect 
1  Project  Engineer 
1  Utility  Engineer 

1  Safety  Engineer 

2  Project  Control  Engineers 
2  Buyers 

1  Quality  Auditor 
1  Project  Lead 


1,985  Document  References 

A  Work  and  Change  Orders 
AElectronic  Corrective  Action  Tracking  System 
(eCATS) 

AMeeting  Minutes 

ARisk  Analysis  Spreadsheets  f  Contingency  & 

ARisk  Mitigation  Plans  j 
AMaturity  Path  to  Premier  Construction  Supplier 
Process 

ABeneficial  Occupancy  Inspectin  and  Close-Out 
Processes 

AeVMS  Data  and  Quad  Reports 
AAs-built  Drawings  and  Plant  Model 
ABuilding  Codes,  Industry  Standards,  and  Regulations 
A  Quality  Audit  Results  and  Corrective  Action  Reports 
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onclusions 


Understanding  the  context  of  Configuration 
Management  and  Process  and  Product  Quality 
Assurance  for  construction  projects  required  the  most 
appraisai  team  deliberation. 


The  organization  is  driven  to  maintain  a  secure  and 
safe  work  place  for  all  site  personnel.  This  has 
created  a  cuiture  of  continuaiiy  improving  work 
processes. 


■  CMMI  is  applicable  to  facilities  maintenance  as  a 
service  and  aiso  to  the  oidest  form  of  engineering, 
construction.  Many  Maturity  Level  3  practices  were 
cieariy  evident  in  the  organization. 
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Questions? 


The  Kansas  City  Plant  manufactures  35  percent  of  NNSA  weapon  products. 


5Cience  int 
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Honeywell  operates  and  manages  the  National  Nuclear  Security  Administration's  Kansas  City  Plant 
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A  RBS  is  the  among  the  top  10  banks  in  the  worid,  mostiy  operating  in  UK, 
Ireland,  US,  Others 


A  RBS  has  development  centres  in  Edinburgh,  London,  India,  Others 

A  iDC  is  the  largest  development  centre  of  RBS  outside  UK 

A  IDC  is  a  1 2  year  old  organization  supporting  multiple  business  lines  T 
Retail  &  Corporate,  Global  Banking,  Insurance 

A  Assessed  at  CMM  level  4,  Certified  to  ISO  9001 , 27001 .  Currently  under 
compliance  review  of  SoX,  processes  aligned  to  CMMI  level  3 

A  Integrated  QA  team  facilitates  delivery  of  implementing  Quality  strategy 
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What  is  QMM 

A  Model  defines  strategies  and  approaches  for  implementing  and 

institutionalizing  Quality  assurance  strategies  in  an  organization  from  Initial 
level  to  continuous  improvement  level 

A  QMM  consists  of  five  maturity  levels  that  reflect  a  degree  of  Quality 
Assurance  (QA)  process  maturity 

A  QMM  (Quality  Maturity  Model)  is  a  proven  framework,  evolved  over  a 
period  of  time  while  deploying  Quality  assurance  practices  in  different 
business  lines/programs  and  identifying  practices  through 
I  pilots 
I  learning 

I  Implementing  best  practices 
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How  can  QMM  Help 

A  QMM  has  been  established  as  a  model  to  support  organizations  meeting 
their  business  objectives 

A  QMM  can  help  define  a  step  by  step  approach  on  improving  and  maturing 
QA  practices  including  quantitative  visibility  and  proactive  improvements 

A  Higher  visibility  of  project  level  QA  and  value  addition  in  overall  delivery 

A  Easy  to  use  and  tailorable  framework 

A  High  level  process  compliance  visible  during  external  assessments/audits 
A  Alignment  of  QA  processes  for  continuous  improvements  at  project  level 
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3MM  5  Maturity  Levels 


€ 

€ 


€ 


Focus  on  continuous 
improvement 


Process  measured 
and  controlled 


QA  Process  characterized  for  the 
organization  and  is  aligned  to 
overall  SDLC 


Process  characterized  for 
supporting  PM  processes 
and  is  localized 

QA  Process  unpredictable 
poorly  controlled,  and 
reactive 
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Level  1  -  Initial 
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A  Level  1 

V  QA  processes  implemented  in  ad  hoc  manner 

T  Reactive  QA  support  required  due  to  problems  at  project  level 

V  Depends  on  what  project  manager  want  (rather  than  what  is  required 
by  the  project)  and  their  view  of  Quality  Assurance 

T  Individual  dependent 

T  Even  project  level  processes  may  not  be  stable 
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Level  2  -  Defined 
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A  Level  2 


T  At  this  level,  projects  select  QA  processes  based  on  their  need  and 
implement  them. 

V  Focus  is  on  having  set  of  QA  processes  which  align  well  with  Project 
management  processes. 

V  Some  project  level  QA  plan  and  measurements  may  be  reported 

V  Project  level  facilitation  is  a  focus  and  reviews  may  are  carried  out,  if 
required. 

T  Lack  of  focus  of  QA  approaches  across  SDLC 

T  No  consistency  across  projects/programs  and  organization  wide 

T  Lack  of  integration  of  project  level  processes  with  organization  wide 
existing  processes 
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Level  3  -  Integrated 


I  At  this  level,  projects  implement  an  organisation  wide  QA  process  (which  is  integrated  with 
other  processes  as  well) 

I  They  have  option  to  tailor  it  based  on  project  specific  need. 

I  At  this  level,  QA  processes  focus  on  ensuring  across  SDLC,  processes  achieve  their  goal. 

I  QA  processes  also  focus  on  ensuring  organisation  wide  understanding  of  processes. 

I  Project  level  reviews  are  planned  along  with  projects  life  cycle  progress  and  focus  is  on  both 
process  &  product  quality  reviews. 

I  Formal  QA  metrics  defined  at  organization  level  are  implemented. 

I  Process  improvements  may  be  initiated  based  on  QA  findings/recommendation 

I  Qrganization  wide  capturing  &  sharing  of  Process  asset  library,  learning  &  suggestions 
Qrganization  wide  Internal  quality  audit  and  independent  reporting  to  management 

I  Consolidation  and  reporting  of  QA  results  at  organization  level 

I  Qrganization  beginning  to  focus  on  implementing  best  practices  from  industry  specific  models 
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4  T  Quantitatively  Managed 


T  At  this  level,  focus  Is  to  manage  the  QA  process  quantitatively  so  that 
project  performance  can  be  provided  adequate  quantitative  visibility 
including  identifying  improvements. 

V  Develop  Balanced  scorecard  for  organization  wide  QA  processes. 

V  Define  control  limits  to  manage  QA  processes  and  publish  an 
organization  wide  process  capability  baseline 

T  Improvements  Identified  based  on  analysis  of  Balanced  scorecard  & 
analysis  of  organization  wide  QA  data 

T  Use  of  statistical  tools  for  improvements  such  as  7  QC  tools,  control 
charts 

V  Establish  Knowledge  management  framework 
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T  Continuous  Improvement 


T  At  this  level,  focus  Is  to  continually  Improve  QA  processes  to  align  with 
ever  improving  delivery  models.  Bring  in  the  proactive  improvement 
element. 

V  Identify  Continuous  improvement  activities  for  QA  at  organization  level 
&  implement  them.  QA  delivers  high  level  of  process  maturity  through 
industry  wide  best  practice  models 

V  Use  of  formal  Improvement  tools  such  as  six  sigma,  lean  management, 
Jurana  methodology,  workout,  for  continuous  improvement 
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QMM  Process  Areas 


I  QA  Facilitation  (Project  Level) 

I  Process  Assurance  (PM  Activities) 

I  QA  Measurements  (Project  Level) 

A  Level  3 

T  Software  Quality  Assurance  (SQA) 

I  Internal  Quality  Audit  (IQA) 

I  QA  Process  Definition  &  tailoring  of  processes 
T  QA  metrics  definition  and  reporting 
T  Process  improvements 
A  Level  4 

T  Quantitative  Management  of  QA  processes 
T  Knowledge  Management  (KM) 

A  Level  5 

T  Causal  Analysis  and  Resolution 
T  Continuous  Improvement 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Level  2 


RBS  -  IDC 


12 


■ 


wm 


L2  T  QA  Facilitation 

A  QA  facilitation  at  project  level 
Identify  and  perform  facilitation 

SQA  facilitation  is  performed  for  supporting  day  to  day  process  need  for  projects 

1 .  Manage  queries  on  processes  by  projects 

2.  Guide  project  manager  in  tailoring  processes  and  templates 

3.  Conduct  training  on  project  specific  QA  processes 

4.  Support  improvements  at  project  level 

5.  Assist  project  for  any  external  certification  and  assessments 
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ess  Assurance  (PM  activities) 

A  Perform  Process  Assurance  focusing  on  (PM  activities) 

Identify  and  perform  process  assurance  for  project  management  related 
activities 

1 .  Review  project  plan  and  project  schedule  for  the  project  at  defined  frequency 

2.  Establish  risk  management  in  the  project 

3.  Support  project  level  tracking  &  reporting 

4.  Take  corrective  action  on  review  findings  as  and  when  required 
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Measurements  (Project  level) 


A  Project  level  QA  measurements  reporting  (schedule,  effort) 


Identify  and  report  QA  measurements  at  the  project  level 

1 .  Define  measurement  to  measure  QA  performance  for  individual  projects 

2.  Report  status  at  project  level 
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ware  Quality  Assurance  (SQA) 


Following  are  the  high  ievei  practices  for  the  process  area: 


A  SQA  Pianning 

I  Plan  for  SQA  activities 
I  Plan  for  SQA  Resourcing 
A  SQA  Activities 

I  SQA  Process  Review 
T  SQA  Product  Review 
A  SQA  Monitoring  and  Controi 
T  Monitor  SQA  Plan 
T  Conduct  Progress  Review 

Click  Here  for  details 
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e  Quality  Assurance  (SQA)  (contde  ) 


A  Plan  for  SQA  activities 


Plan  for  management  of  project  SQA  activities 

SQA  prepares  a  periodic  schedule  of  the  planned  SQA  activities  The  schedule 
covers  the  following  tasks: 

»  Process  reviews 

»  SQA  facilitation 

»  Document  reviews 

1 .  Identify  all  SQA  activities  for  the  period  with  planned  effort 

2.  Establish  a  mechanism  to  take  input  and  agreement  from  project  manager  for 
SQA  plan.  Align  with  project  plan 

3.  Update  plan  on  a  defined  frequency 
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e  Quality  Assurance  (SQA)  (contde  ) 


A  Plan  for  SQA  Resourcing 


Establish  and  maintain  the  SQA  resource 

Better  planning  and  identification  of  SQA  resources  in  advance  help  in  supporting 
the  projects  better  and  avoid  and  surprises. 


1 .  Establish  and  maintain  an  organizational  policy  for  planning  and  performing 
the  SQA  process 

2.  Provide  adequate  resources  for  performing  the  SQA  process 

3.  Assign  responsibility  and  authority  for  performing  the  SQA  process 

4.  Train  the  people  performing  or  supporting  the  SQA  process  as  needed 

5.  Collect  historical  data  on  SQA  effort  and  the  activities  performed 

A  This  data  act  as  a  basis  for  identifying  the  average  SQA  effort  which  is  required  for 
forecasting  the  SQA  resources. 
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e  Quality  Assurance  (SQA)  (contde  ) 


SQA  Process  Review 


Objectively  evaluate  the  designated  performed  SDLC  processes  against 
the  applicable  process  descriptions,  standards,  and  procedures. 

1 .  Establish  and  nnaintain  clearly  stated  criteria  for  the  evaluations. 

A  What  will  be  evaluated 

A  When  or  how  often  a  process  will  be  evaluated 
A  How  the  evaluation  will  be  conducted 
A  Who  must  be  involved  in  the  evaluation 

2.  Use  the  stated  criteria  to  evaluate  performed  processes  for  adherence  to 
process  descriptions,  standards,  and  procedures. 

3.  Identify  each  noncompliance  found  during  the  evaluation. 

4.  Identify  lessons  learned  that  could  improve  processes  for  future  products  and 
services. 
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e  Quality  Assurance  (SQA)  (contde  ) 


A  SQA  Product  Review 


Objectively  evaluate  the  designated  work  products  and  services  against  the 
applicable  process  descriptions,  standards,  and  procedures. 

1 .  Select  work  products  to  be  evaluated,  based  on  documented  sampling  criteria  if 
sampling  is  used. 

2.  Establish  and  maintain  clearly  stated  criteria  for  the  evaluation  of  work  products. 

3.  Use  the  stated  criteria  during  the  evaluations  of  work  products. 

4.  Evaluate  work  products  before  they  are  delivered  to  the  customer. 

5.  Evaluate  work  products  at  selected  milestones  in  their  development. 

6.  Perform  in-progress  or  incremental  evaluations  of  work  products  and  services 
against  process  descriptions,  standards,  and  procedures. 

7.  Identify  each  case  of  noncompliance  found  during  the  evaluations. 
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e  Quality  Assurance  (SQA)  (contde  ) 


A  Monitor  SQA  Plan 


Monitor  commitments  against  those  identified  in  the  SQA  pian. 

1 .  Regularly  review  commitments  (both  external  and  internal). 

2.  Identify  commitments  that  have  not  been  satisfied  or  that  are  at  significant  risk 
of  not  being  satisfied. 

3.  Document  the  results  of  the  commitment  reviews. 
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e  Quality  Assurance  (SQA)  (contde  ) 

A  Conduct  Progress  Review 

Periodically  review  the  QAG  progress,  performance,  and  issues. 

1 .  Review  of  QA  group  progress  on  the  plan  at  defined  frequency  (weekly, 
monthly)  to  track  performance  of  plans,  issues/  findings  raised  during  reviews 
and  their  status/  escalations. 

2.  Share  summary  status  with  stakeholder  management 
Typical  Work  Products 

I  QAG  task  list 
I  Project  Status  Review 
I  QA  group  metrics 
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Internal  Quality  Audit  (IQA) 


Following  are  the  high  ievei  practices  for  the  process  area: 
A  Pianning  IQA 
A  Conducting  IQA 
A  Monitoring  &  ciosing  IQA 


Click  Here  for  details 
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irnal  Quality  Audit  (IQA)  (contde  ) 


A  Planning  IQA 


Establish  a  high-level  yearly  IQA  plan. 

1 .  Identify  the  various  sources  of  input  to  the  plan.  The  various  sources  can  be: 

A  Inputs  from  Senior  Management 
A  Inputs  from  project/program  milestones 

A  Input  from  previous  year®  Internal  Quality  Audit  reports/external  audit/  assessment 
plans 

A  Inputs  from  SQA  Plan 

2.  Develop  the  plan  at  the  start  of  the  year 

3.  Review  and  update  the  plan 
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irnal  Quality  Audit  (IQA)  (contde  ) 


A  Planning  IQA  (contd..) 


Establish  and  maintain  monthly  IQA  schedule  as  per  the  defined  audit 
coverage  criteria 

1 .  Develop  and  define  the  audit  coverage  criteria. 

The  coverage  for  the  projects  can  be  based  on  various  factors  like  size,  compiexity,  iSQA 
findings.  Support  groups  can  also  be  identified  to  be  covered  at  a  specified  frequency 
(typicaiiy  once  in  quarter  ) 

2.  Develop  monthly  IQA  schedule  and  circulate  it  to  all  key  stakeholders  (auditor 
and  auditee)  for  their  acceptance 

3.  Make  available  the  plan  at  a  central  repository  for  all  stakeholders 
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A  Conducting  IQA 


Perform  audit  as  per  the  schedule. 

1 .  The  Internal  Audit  is  conducted  as  per  the  published  processes  used  for 
carrying  out  the  activities. 

2.  Project  Manager  is  responsible  to  show  the  evidences  of  process 
documentation. 

3.  Internal  auditor(s)  will  record  the  findings  in  audit  note  sheet  and  get  it  signed  off 
from  auditee. 

4.  Based  on  the  findings,  the  auditor  will  prepare  the  internal  audit  report 

5.  The  approved  internal  audit  report  is  sent  to  Project  Manager  for  filling  the 
corrective  and  preventive  actions. 

Typical  Work  Products 
1 .  IQA  report 
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irnal  Quality  Audit  (IQA)  (contde  ) 


Monitor  the  IQA  progress  against  the  planned  schedule  and  follow  up  for 
closure  of  non-conformances. 

1 .  Monitor  IQA  progress  against  the  schedule. 

A  Progress  monitoring  typically  includes  the  following: 

A  Periodically  measuring  the  actual  completion  of  activities  and  milestones 
A  Identifying  significant  deviations  from  the  schedule  estimates  in  the  IQA  plan 

2.  Document  the  significant  deviations  in  the  project  planning  parameters. 

3.  Follow  up  on  closure  of  identified  non-conformances  and  observations 

4.  Perform  escalation  in  a  timely  manner  to  avoid  process  breakthrough  situation 
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T  QA  Process  Definition 

Following  are  the  high  level  practices  for  the  process  area: 
A  Establish  Quality  Group  Process  Assets 
A  Establish  Tailoring  Criteria  and  Guidelines 
A  Establish  the  Quality  Group’s  Process  Asset  Library 
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A  Establish  Quality  Group  Process  Assets 

Establish  quality  group  process  assets. 

1 .  Decompose  each  standard  process  into  constituent  process  elements  to 
the  detail  needed  to  understand  and  describe  the  process. 

2.  Specify  the  critical  attributes  of  each  process  element. 

3.  Ensure  that  there  is  appropriate  integration  among  the  processes  that  are 
included  in  the  organization®  set  of  standard  processes. 

4.  Document  the  organization's  set  of  standard  processes. 

5.  Conduct  peer  reviews  on  the  organization's  set  of  standard  processes. 

6.  Revise  the  organization's  set  of  standard  processes  as  necessary. 
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A  Establish  Tailoring  Criteria  and  Guidelines 

Establish  and  maintain  the  tailoring  criteria  and  guidelines  for  the  quality 
group’s  set  of  standard  processes. 

The  tailoring  criteria  and  guidelines  describe  the  following: 

1 .  Mandatory  requirements  that  must  be  satisfied  by  the  defined  processes 

2.  Options  that  can  be  exercised  and  criteria  for  selecting  among  the  options 

3.  Procedures  that  must  be  followed  in  performing  and  documenting  process 


tailoring 


Typical  Work  Products 
1.  Tailoring  guidelines 
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A  Establish  Quality  Group  Process  Asset  Library 

Establish  and  maintain  the  process  asset  library. 

1 .  Design  and  implement  the  quality  group®  process  asset  library,  including  the 
library  structure  and  support  environment. 

2.  Specify  the  criteria  for  including  items  in  the  library. 

3.  Specify  the  procedures  for  storing  and  retrieving  items. 

4.  Enter  the  selected  items  into  the  library  and  catalog  them  for  easy  reference  and 
retrieval. 

5.  Make  the  items  available  for  use  by  the  projects. 

6.  Periodically  review  the  use  of  each  item  and  use  the  results  to  maintain  the 
library  contents 
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Metrics  Reporting 


Following  are  the  high  level  practices  for  the  process  area: 


A  Establish  a  metrics  framework 
A  Report  metrics 


Click  Here  for  details 
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A  Establish  a  metrics  framework 

Establish  a  mechanism  for  metrics  definition  for  QA  group. 

1 .  Define  the  various  measures  required  for  QA  group 

2.  Identify  the  data  collection  mechanism  and  consolidation 

3.  Identify  the  tailorable  aspects  of  metrics  if  any 

4.  Integrate  the  metrics  as  part  of  overall  QA  processes 

5.  Tolerance  for  metrics  to  be  defined  and  used  for  tracking  and  reporting 
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A  Report  metrics 

Establish  a  mechanism  for  metrics  reporting  at  QA  group  level  and 
organization  level. 

1 .  Consolidation  of  data  in  a  central  repository 

2.  Report  the  metrics  at  identified  frequency 

3.  Reporting  of  metrics  data  through  QA  group  reports  and  organization  wide 


reports 
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A  Perform  process  improvement 

Establish  a  mechanism  for  performing  process  improvement  arising  out  of 
project  recommendations/QA  findings. 

1 .  Define  the  mechanism  of  receiving/identifying  QA  findings  /  recommendations  / 
suggestions 

2.  Perform  impact  analysis  and  identify  the  necessary  process  changes 

3.  Make  changes  to  the  process  and  sent  it  for  review 

4.  Approved  improvement  is  incorporated  into  organization  wide  QA  processes 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


RBS  -  IDC 


35 


■ 


wm 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 

and  Expanded  Features 


Quantitative  Management 


Following  are  the  high  level  practices  for  the  process  area: 
A  Establish  measurement  objectives 
A  Specify  Measures 

A  Specify  Data  coilection  and  Storage  procedures 
A  Specify  Anaiysis  procedures 
A  identify  improvements 
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Quantitative  Management 

A  Establish  measurement  objectives 

Measurement  objectives  documents  the  purpose  for  which 
measurement  and  anaiysis  are  done,  and  specify  the  kind  of  actions 
that  may  be  taken  based  on  the  resuits  of  data  anaiyses. 

1 .  Set  up  QA  group  measurement  objectives  aligned  to  measure  performance 
against  4  quadrants  of  Balanced  Scorecard 

2.  The  sources  for  measurement  objectives  may  be  management,  technical, 
project,  product,  or  process  implementation  needs. 

3.  Example  of  measurement  objectives  include  the  following: 

A  Findings/Non-conformances  closure  cycle  time 

A  Cycle  time/Benefits  of  implementation  of  learning/suggestion/best  practices 
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Specify  Measures 


Measurement  objectives  driven  from  BSC  are  refined  into  precise,  quantifiable  measures. 

1 .  Identify  measures  for  each  of  the  4  quadrants  (Delivery,  People,  Financial,  Customer) 

2.  Measures  may  be  either  fbaseoor  hderived.oData  for  base  measures  are  obtained  by  direct 
measurement.  Data  for  derived  measures  come  from  other  data,  typically  by  combining  two  or  more 
base  measures. 

3.  Establish  goal  and  thresholds  for  the  defined  BSC  measures 

I  Goals  and  thresholds  may  be  either  developed  using  analysis  of  historical  data  (through  PCB)  or  through 
management  targets  /  priorities 

4.  Examples  of  commonly  used  base  measures  include  the  following: 

A  Average  non-conformance  closure  cycle  time 

A  IQA  compliance  with  monthly  schedule 
A  Overall  staff  retention 

5.  Examples  of  commonly  used  derived  measures  include  the  following: 

A  No.  of  SQA  findings  per  project  per  review 

A  Average  SQA  effort /project/month 
A  No.  of  training  hours  per  year  per  member 
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A  Specify  data  collection  and  storage  procedures 

Explicit  specification  of  collection  methods  helps  ensure  that  the  right 
data  are  collected  properly.  It  may  also  aid  in  further  clarifying 
information  needs  and  measurement  objectives. 

1 .  Identify  existing  sources  of  data  that  are  generated  from  current  work  products, 
processes,  or  transactions. 

2.  Identify  measures  for  which  data  are  needed,  but  are  not  currently  available. 

3.  Specify  how  to  collect  and  store  the  data  for  each  required  measure. 
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antitative  Management  (contde  ) 

A  Specify  Analysis  procedures 

Specifying  the  anaiysis  procedures  in  advance  ensures  that  appropriate 
anaiyses  wiii  be  conducted  and  reported  to  address  the  documented 
measurement  objectives  (and  thereby  the  information  needs  and 
objectives  on  which  they  are  based).  This  approach  aiso  provides  a 
check  that  the  necessary  data  wiii  in  fact  be  coiiected. 

1 .  Following  are  the  analysis  mechanisms  used: 

A  Process  Capability  Baseline 
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antitative  Management  (contde  ) 


aseline 


1 .  PCB  represents  performance  of  various  QA  group  processes  in  the  organization  in 
quantitative  terms.  It  also  forms  as  a  basis  for  predicting  the  behavior  of  the 
processes  in  near  future  assuming  that  similar  kind  of  work  will  be  performed. 

2.  Measurements  and  metrics  related  to  QA  group  which  have  to  be  baselined  are 
identified  and  prioritized. 

3.  Prepare  an  analysis  report  using  appropriate  statistical  techniques 

4.  Use  analysis  of  data  to  set  /  refine  goal  and  thresholds  for  measures  in  BSC 

5.  Example  of  high  level  metrics  which  can  be  baseline  are: 

A  Average  SQA  effort/project/month:  It  helps  in  forecasting  the  actual  QA  resource 
requirement  for  the  future  projects. 

A  Number  of  SQA  findings/project/review:  It  helps  in  identifying  the  process  compliance 
in  the  projects. 
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antitative  Management  (contde  ) 

A  Identify  improvements 

Improvements  are  formally  identified  from  the  data  analysis  performed. 

1 .  Improvements  are  identified  from  Balanced  scorecard,  PCB  using 
statistical  tools  like  7  QC  tools. 

2.  Analyze  the  organization's  set  of  standard  processes  to  determine  areas 
where  improvements  would  be  most  helpful 

3.  Pilot  improvements 

4.  Select  improvements  for  deployment 
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Knowledge  Management 


A  Establish  Knowledge  Management 


Set  up  knowledge  management  framework 


1 .  Identify  an  appropriate  tool  to  deploy  the  KM  framework 

T  Example  of  tools  can  be  workflow  systems  (Lotus  Notes),  Internet  based 
applications,  excel  based  tool. 

2.  Set  up  KM  framework  to  capture  knowledge  at  various  part  of  SDLC 
(e.g.  Best  practices,  learning) 

3.  Organize  the  received  assets 

4.  Share  the  knowledge  through  documents 

5.  Use/reuse  the  assets 

6.  Identify  improvements  if  any 
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> Rating  &  evaluation 
>Assets  improvement. 

>Benefits  realized 
>Lessons  learnt 
>KM  Process/Tool  Implovement 
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>Search  for  examples, 
query  resolutions 
>Discussion  forum/Practice  Communities 

> Usage  of  available  assets. 

>Knowledge  Sharing 


I 
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> Project/Functional  group  Experiences 

>Queries/Solutions 

>Tips  &Tricks 

>Best  Practices  (Industry) 

>Post  Project  Review 


>  Categorization 

>  Review  &  approve 


>  Document 

>  Distribute 
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ausal  Analysis  &  Resolution 


Following  are  the  high  level  practices  for  the  process  area: 
A  Determine  Causes  of  Non-conformances 
A  Analyze  Causes 
A  Implement  the  Action  Proposals 


Click  Here  for  details 
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Determine  Causes  of  Non-conformance 


Root  causes  of  non-conformances  and  other  findings  are  systematically 
determined.. 

1 .  Gather  relevant  non-conformance  and  finding  data. 

Examples  of  relevant  non-conformance  data  may  include  the  following: 

I  Internal  quality  audit  non-conformances 
V  QA  review  findings 

2.  Determine  which  non-conformances  and  other  findings  will  be  analyzed  further. 

Examples  of  methods  for  selecting  defects  and  other  problems  include  the 
following: 

A  Pareto  analysis 
A  Histograms 
A  Control  Charts 
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A  Analyze  Causes 


1. 

2. 

3. 


Root  causes  of  non-conformances  and  other  findings  are  systematicaiiy 
determined.. 

Conduct  causal  analysis  with  the  people  who  are  responsible  for  performing  the 
task. 

Analyze  selected  non-conformances  and  other  findings  to  determine  their  root 
causes. 

Propose  and  document  actions  that  need  to  be  taken  to  prevent  the  future 
occurrence  of  similar  non-conformances  and  other  findings. 


1. 

2. 

3. 
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A  Implement  the  action  proposals 

Implement  the  selected  action  proposals  that  were  developed  in  causal 
analysis. 

1 .  Analyze  the  action  proposals  and  determine  their  priorities 

2.  Select  the  action  proposals  that  will  be  implemented. 

3.  Create  action  items  for  implementing  the  action  proposals 
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Continuous  Improvement 

A  Continuously  improve  the  processes 

Identify  and  continuously  deploy  the  new  improved  processes  /  tools  / 
methods 

1 .  Identify  Cl  initiatives  to  achieve  organization  objectives/goals  identified  in 
Balanced  Scorecard 

2.  Take  up  Cl  projects  using  appropriate  tools  such  as  six  sigma,  lean 
management,  work  out 

3.  Encourage  cross  functional  team  based  Cl 

4.  Review  performance  of  initiatives  /  Cl  projects 

5.  Report  status  &  benefits  to  management 
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Summary 

QA  Process  Maturity 

Project  /  Program  Process  Maturity 

Level  2 

AQA  focused  on  facilitating  project 
management  processes 

AQA  review  PM  artifacts 

AQA  reporting  at  project  level 

AMatured  PM  processes  for  projects 

ASetter  quality  PM  deliverables 

ASetter  insight  into  regular  project  monitoring  & 
tracking 

Level  3 

Aqa  process  focused  on  establishing  process 
asset  library,  initial  metrics  framework 

AiQA  is  established 

ASQA  support  for  entire  SDLC 

ASharing  of  learning  /  best  practices  across 
projects 

ASQA  support  for  entire  SDLC  leading  to  improve 
engineering  deliverables 

Alhird  party  view  of  project  through  IQA 

APro-active  identification  of  findings 

Level  4 

AKnowledge  Management 

AOuantitative  Management 

AOuantitative  visibility  into  QA  process 
management  through  BSC 

AEnd  to  end  active  repository  for  project  learning, 
documents,  tips  &  tricks. 

Level  5 

AHausal  Analysis  &  Resolution 

ADontinuous  improvement 

AOecrease  in  in  process  and  post  delivery  defects 
using  identified  Cl  tools 

Aimproved  budget  using  innovative  techniques  for 
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Re-introducing  the  CMMI  for 
Services  Constellation 


CMMI  Technology  Conference  and  User  Group 

November  12-15,  2007 

Craig  R.  Hollenbach 
Northrop  Grumman  Corporation 

Brandon  Buteau 
Northrop  Grumman  Corporation 

Drew  Allison 

Systems  and  Software  Consortium  Inc. 

Frank  Niessink 
DNV-CIBIT 
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•  CMMI-SVCNews 

•  Overview  of  the  draft  CMMI  for  Services  (CMMI-SVC) 

•  What  is  the  CMMI? 

•  Why  is  the  CMMI-SVC  needed? 

•  How  are  services  different? 

•  What  is  the  basis  for  the  CMMI-SVC  model? 

•  What  is  the  scope  and  content  of  the  CMMI-SVC? 

•  Feedback  to  date 

•  What  was  the  result  of  the  expert  review? 

•  What  was  the  experience  of  the  pilot  projects? 

•  Next  Steps 

•  What  is  the  schedule? 

•  How  can  I  participate? 
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CMMI 


•  There  was  a  serious  concern  that  concurrent  development  of 
the  CMMI-ACQ  and  CMMI-SVC  models  would  stress  the  SEI 
resources  needed  to  deliver  the  CMMI-ACQ  model  on  time. 
Now  that  CMMI-ACQ  is  almost  released,  the  SEI  resources 
are  available  to  go  forward  with  the  CMMI-SVC  development. 


ID 

Task  Name 

1 

CMMI-DEV,  v1.2 

2 

CMMI-ACQ  (OM) 

3 

CMMI-ACQ.  v1 .2 

4 

CMMI-SVC.  vO. 5 

5 

CMMI-SVC.  vO.5  review 

6 

CMMI-SVC.  v1 .2 
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CMMI 


•  A  conceptual  framework  for  structuring,  understanding,  and 
evaluating  the  capability  and  maturity  of  an  organization® 
processes 

•  more  than  a  laundry  list  of  best  practices 

•  more  than  a  coilection  of  benchmarks  and  metrics 

•  A  tool  that  enables  meaningful,  in-depth  organizational 
assessment 

•  internally 

•  externally 

•  A  map  that  guides  practical  process  improvement  and 
institutionalizes  it 

•  How  to  you  get  from  here  to  there  and  stay  there? 
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•  The  CMM  Integration®'^  (CMMI)  of  multiple  CMMs  into  a 
single  unified  framework 

EIA  Interim  Standard  731, 
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CMMK 


CMMI-DEV 

provides  guidance 
for  measuring, 
monitoring,  and 
managing 
development 
processes 

CMMI-SVC 

provides  guidance  for 
those  providing 
services  within 
organizations  and  to 
external  customers 

\  \  lx 

Core 
Model 
Foundation 


CMMI-ACQ 

provides  guidance 
to  enable 
informed  and 
decisive 
acquisition 
leadership 


Courtesy  of  the  SEI 
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•  Customer  discontent 

•  Service  society 

•  Legislation 

•  Government  and  industry 
trends 
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•  Services  form  a  distinctive  category  of  products 

•  A  service  is  an  intangible,  non-storable  product 

•  What  makes  a  service  intangibie  or  non-storable? 

•  Customer  desires  a  situation  or  state  (e.g.,  to  have  high  network 
availability)  rather  than  a  tangible  artifact 

•  Provider  delivers  value  without  independent,  unrestricted  means  of 
generating/employing  that  value  by  the  customer  (e.g.,  leasing  vehicles) 

•  Product  delivery  requires  continuing  application  of  labor  (e.g.,  operation 
of  a  facility) 

•  Services  imply  customer/provider  relationships  governed  by 

service  agreements 

•  Service  and  non-service  products  may  be  delivered  as  part  of  a  single 

agreement  (e.g.,  training  that  includes  hardcopy  materials) 

•  Services  are  often  delivered  via  the  operation  of  a  service  system 
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•  A  necessary  concept  for  understanding  the  effective 
delivery  of  services 

•  An  integrated  and  interdependent  combination  of 
processes,  resources,  and  people  that  satisfies  service 
requirements. 

•  Portions  are  not  delivered  to  the  customer  or  end-user  as 
part  of  service  delivery 

•  Portions  may  remain  owned  by  the  customer  or  end-user 
before  service  delivery  begins  and  after  service  delivery 
ends. 

•  Encompasses  everything  required  for  service  delivery, 
including  work  products,  processes,  infrastructure, 
consumables,  and  customer  resources. 
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•  Covers  practices  required  to  manage,  establish,  and  deliver 
services,  in  four  process  area  categories 

•  Project  (service)  management 

•  Process  management 

•  Service  support 

•  Service  estabiishment  and  delivery 

•  Intended  to  match  the  scope  of  the  definition  of  services 

•  Broad  applicability  to  a  range  of  service  domains 

•  information  technoiogy,  engineering,  defense,  transportation, 
finance,  heaith  care 

•  Staff  augmentation  services  need  careful  consideration 

•  How  do  you  evaiuate  process  improvement  for  processes  over 
which  you  have  no  controi? 
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Process  Management 

Organizational  Innovation  and 
Deployment  (OID) 

Organizational  Process  Definition  (OPD) 
Organizational  Process  Focus  (OPF) 

Organizational  Process  Performance 
(OPP) 

Organizational  Service  Management 
(OSM) 

Organizational  Training  (OT) 

Service  Support 

Causal  Analysis  and  Resolution  (CAR) 
Configuration  Management  (CM) 
Decision  Analysis  and  Resolution  (DAR) 
Measurement  and  Analysis  (MA) 
Problem  Management  (PRM) 

Process  and  Product  Quality  Assurance 
(PPQA) 


Service  Establishment  and  Delivery 

•  Incident  and  Request  Management 
(IRM) 

•  Service  Delivery  (SD) 

•  Service  System  Development  (SSD) 

•  Service  Transition  (ST) 

Project  Management 

•  Capacity  and  A  vai lability  Management 

•  Integrated  Project  Management  (I PM) 

•  Project  Monitoring  and  Control  (PMC) 

•  Project  Planning  (PP) 

•  Requirements  Management  (REQM) 

•  Risk  Management  (RSKM) 

•  Quantitative  Project  Management  (QPM) 

•  Service  Continuity  (SCON) 

•  Supplier  Agreement  Management  (SAM) 
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Process  Area 

Maturity  Level 

Specific  Goais/ 
Practices 

Capability  and  Availability  Management  (CAM) 

3 

2/6 

Incident  and  Request  Management  (IRM) 

2 

2/6 

Organizational  Service  Management  (OSM)* 

3 

2/7 

Problem  Management  (PRM) 

3 

2/7 

Service  Continuity  (SCON)* 

3 

3/10 

Service  Delivery  (SD) 

3 

2/7 

Service  System  Development  (SSD)  * 

3 

3/12 

Service  Transition  (ST) 

3 

3/12 

*  optional  process  areas  (independent  named  additions) 
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•  Incident  and  Request  Management 

•  To  ensure  the  timely  resolution  of  requests  for  service 
and  incidents  that  occur  during  service  delivery 

•  Requirements  Management 

•  Extended  from  the  Core  Model  Foundation  with  an 
additional  goal 

•  To  include  the  establishment  and  maintenance  of  written 
agreements  between  service  providers  and  customers 
on  service  requirements  and  service  levels. 

•  Six  other  level  2  PAs  from  the  CMF 
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•  Capacity  and  Availability  Management 

•  To  plan  and  monitor  the  effective  provision  of  resources 
to  support  service  requirements 

•  Problem  Management 

•  To  prevent  incidents  from  recurring  by  identifying  and 
addressing  underlying  causes  of  incidents 

•  Service  Delivery 

•  To  deliver  services  in  accordance  with  service 
agreements 

•  Service  Transition 

•  To  deploy  new  or  significantly  changed  service  systems 
while  managing  their  effect  on  ongoing  service  delivery 
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•  Organizational  Service  Management 

•  To  establish  and  maintain  standard  services  that  ensure 
the  satisfaction  of  the  organization's  customer  base 

•  Service  Continuity  Management 

•  To  establish  and  maintain  contingency  plans  for 
continuity  of  agreed  services  during  and  following  any 
significant  disruption  of  normal  operations 

•  Service  System  Development 

•  To  analyze,  design,  develop,  integrate,  and  test  service 
systems  to  satisfy  existing  or  anticipated  service 
agreements 
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•  An  expert  review  was  held  Jan  23  -  Mar  23,  2007 

•  500+  reviewers,  representing: 

•  50  companies, 

•  14  DoD  organizations, 

•  4  academic  institutions,  and 

•  7  professionai,  governmentai,  or  research  centers 

•  Reviewers  included  SEI  transition  partners 

•  Response  showed  strong  interest  in  CMMI-SVC 

•  900+  change  requests  compares  favorably  to  those 
received  for  CMMI-DEV 

•  50  survey  responses  to  architectural  questions 
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•  Reviews  commented  most  on  CMM-SVC  architecture  &  Common 
Model  Foundation  material 

•  CRs  were  distributed  equally  among  categories  related  to  SVC  PAs 

•  CMMI-SVC  team  has  analyzed  all  architectural  CRs;  most  have  a 
proposed  resolution 

•  CRs  showed  excellent  depth  of  insight  and  rich  informatve  content 
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•  The  service  practices  that  are  covered  in  CMMI-SVC  will  enable  service  organizations  to  provide  more 
effective  support  to  their  customers. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

78.9% 

8.8% 

12.3% 

•  The  material  in  CMMI-SVC  yields  a  useful  adaptation  of  CMMI  best  practices  as  they  relate  to  service 
deployment. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

66.7% 

14.0% 

15.8% 

•  CMMI-SVC  does  not  impose  constraints  (derived  from  the  needs  of  a  specific  service  or  market 

segment)  that  would  limit  or  prevent  other  organizations  from  adapting  the  model  to  their  own  specific 
needs. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

55.6% 

29.6% 

27.8% 

•  The  CMMI-SVC  is  easy  to  understand  and  apply  to  a  service  organization. 


Strongly  Agree  or  Agree 

Neutral 

Disagree  or  Strongly  Disagree 

42.8% 

27.8% 

29.6% 
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•  Planned  pilots  were  postponed 

•  CMMI-SVC  participating  companies  piloted  the  model  internally 

•  Characteristics  of  the  piloted  organizations: 

•  Most  had  implemented  CMMI-DEV 

•  Some  had  separate  ITIL  and  ISO  20000  initiatives 

•  Most  are  moving  towards  integration  under  CMMI  umbrella 

•  The  pilots  represented  the  following  service  domains: 


Company 

Service  Domains 

SSCI 

IT  Application  Operations  &  Support 

DNV-CIBIT 

Banking 

Northrop  Grumman 

Logistics,  HR,  IT,  Applications  O&M 
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•  Improved  quality  of  services 

•  Encouraged  a  discipiined  culture  for  service  management 

•  Better  management  visibility  into  services 

•  Fewer  surprises 

•  Fosters  process  improvement 

•  Less  Interpretation  issues  (&  appraisai  expense)  than  with  CMMi-DEV 

•  Appiying  a  CMMI  process  to  the  services  brought  credibility  and  buy-in 
from  stakehoiders 

•  increased  sharing  between  deveiopment  and  services  communities 

•  Common  processes 

•  Standard  terminology 

•  Integrated  process  improvement  standards  and  models 

•  Encouraged  end-to-end  lifecycle  process  approach  helping  to  identify 
service  requirements,  ease  deployment  issues,  reduce  stove-piped 
groups,  and  improve  efficiencies  of  support-reiated  groups  (iT 

Appi  cations) 
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•  Obtaining  funding  in  environments  that  are  primarily  LOE-based 

•  Differences  in  terminology  between  development  and  services 

•  Terms  like  fProjectoffunding  period),  fProductofservice),  fWork  Productq 
fProduct  Componentq  fRequirementb 

•  Interpreting  CMMIo  fprojecteterm  for  services 

•  No  standard  life-cycle  definition  for  services 

•  Instilling  project  management  culture  in  services 

•  Weak  in  using  requirements  for  planning  and  negotiating  resources  and 
activities 

•  Ownership  of  service  system  components  not  as  clear 

•  Release  management  and  deployment  to  non-standardized,  constantly 
changing  environments 

•  Finding  CMMI-knowledgeable  individuals  who  also  know  services 

•  Integrating  process  groups  and  assets 

•  Services  where  customer  and  provider  share  resources  and  processes 

•  Staff  augmentation 
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•  CMMI-SVC  team  will  meet  to  review  additional 
requirements  and  re-plan  remaining  work  (early  Nov) 

•  Detailed  schedule  is  pending 

•  A  preliminary  estimate  for  release  of  CMMI-SVC,  v1 .2  is 
4**^  quarter  2008 


ID 

Task  Name 

2005 

2006 

2007 

2008 

Qtrl  Qtr2  Qtr  3  Qtr  4 

Qtrl  Qtr  2  Qtr  3  Qtr  4 

Qtrl  Qtr  2  Qtr  3 

Qtr  4 

Qtrl  Qtr  2  Qtr  3  Qtr  4 

4 

CMMI-SVC,  vO.5 

5 

CMMI-SVC,  vO.5  review 

6 

CMMI-SVC,  v1 .2 
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•  Get  more  information  about  CMMI-SVC 

•  CMMI  web  page  -  http://www.sei.crnu.edu/crnnni/ 

•  CMMI  for  Services  Public  Workspace 
(http://bscw.sei.cnnu.edU/bscw/bscw.cai/0/424939)  contains: 

•  Draft  CMMI-SVC  model,  vO.5 

•  Information  on  joining  CMMI-SVC  information  email  list 

•  Review  draft  CMMI-SVC  release 

•  If  already  experienced  in  CMMI,  consider  piloting  the  model 

•  Other  opportunities  may  exist  as  a  result  of  the  CMMI-SVC 
re-planning  effort;  watch  CMMI-SVC  public  workspace  for 
updates 
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•  CMMI  -  http://www.sei.cmu.edu/cmmi/cmmi.html 

•  ITIL  -  http://www.oac.aov.uk/index.asp?id=2261 

•  itSMF  -  http://www.itsmf.com/ 

•  BS  1 5000  -  http://www.bs1 5000.ora.uk/ 

•  COBIT  -  http://www.isaca.ora/ 

•  ITSCMM  -  http://www.itservicecmm.ora/ 

•  Interpreting  Capability  Maturity  Model  Integration  (CMMI)  for 
Operational  Organizations,  Brian  P.  Gallagher,  Technical  Note, 
CMU/SEI-2002-TN-006,  April  2002 

•  Interpreting  Capability  Maturity  Model  Integration  (CMMI)  for  Service 
Organizations  T  a  Systems  Engineering  and  Integration  Services 
Example,  Mary  Anne  Herndon,  SAIC,  et  al.  Technical  Note,  CMU/SEI- 
2003-TN-005,  November  2003 

•  Services  CMMI  Public  Website  - 
http://bscw.sei.cmu.edU/bscw/bscw.cai/0/424939 
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•  Development  Team 

•  Craig  Hollenbach  (Northrop  Grumman)  -  Lead 

•  Roy  Porter  (Northrop  Grumman) 

•  Brandon  Buteau  (Northrop  Grumman) 

•  Lynn  Penn  (Lockheed  Martin) 

•  Frank  Niessink  (DNV/CIBIT) 

•  Jerry  Simpson  (SAIC) 

•  Drew  Allison  (SSCI) 

•  Eileen  Forrester  (SEI) 

•  Barbara  Tyson  (SEI) 

•  Eileen  Clark  (SRA) 

•  other  contributors 

•  Jeff  Zeidler  (Boeing) 

•  Rich  Raphael  (Mitre) 

•  Joanne  04.eary  (SEI) 
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1.  The  service  practices  that  are  covered  in  CMMI-SVC  will  enable  service  organizations  to 
provide  more  effective  support  to  their  customers. 

2.  The  material  in  CMMI-SVC  yields  a  useful  adaptation  of  CMMI  best  practices  as  they 
relate  to  service  deployment. 

3.  The  CMMI-SVC  appropriately  uses  the  CMMI  framework. 

4.  CMMI-SVC  includes  process  areas  that  must  be  satisfied  for  process  improvement  and 
institutionalization. 

5.  CMMI-SVC  does  not  impose  constraints  (derived  from  the  needs  of  a  specific  service  or 
market  segment)  that  would  limit  or  prevent  other  organizations  from  adapting  the  model 
to  their  own  specific  needs. 

6.  The  CMMI-SVC  is  easy  to  understand  and  apply  to  a  service  organization. 

7.  The  process  areas  in  CMMI-SVC  cover  all  significant  service-specific  requirements  and 
effectively  reflect  activities  that  a  service  organization  should  be  accomplishing. 

8.  Additions  and  amplifications  that  exist  in  other  models  and  are  also  used  within  the  CMMI- 
SVC  constellation  are  appropriate. 

9.  Notes  and  examples  in  CMMI-SVC  clearly  apply  to  service  organizations  and  meet  their 
specific  needs. 

10.  References  in  PAs  to  related  process  areas  are  clear  and  consistently  applied. 


26 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


rade  to 
and  Expanded  Features 


suiib  lo  General  Survey 


CMMI 


Survey  Responses 


□  Agree  or  Strongly  Agree 

□  Neutral 

□  Disagree  or  Strongly 
Disagree 


27 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


a/id  Expanded  Features 


sswea  Questions 


CMMI 


A.  Problem  management  practices  that  are  common  within  the  service  industry  are  appropriately 
addressed  in  the  process  area  Problem  Management  and  are  distinguished  from  the  practices  in  the 
Causal  Analysis  and  Resolution  process  area. 

B.  The  Project  Management  category  is  the  most  appropriate  classification  for  the  Service  Continuity 
Management  and  Capacity  and  Availability  Management  process  areas. 

c.  The  Process  Management  category  is  the  most  appropriate  classification  for  the  Organizational 
Service  Management  process  area 

D.  The  practices  within  the  Service  Continuity  process  area  should  build  upon  the  practices  within  the 
Risk  Management  process  area  similar  to  the  manner  in  which  the  Integrated  Project  Management 
process  area  builds  upon  maturity  level  2  project  management  practices. 

E.  The  Service  System  Development  process  area  must  be  required  for  an  organization  to  be  a  mature 
service  organization. 

F.  The  specific  practices  in  the  Service  System  Development  process  areas  are  presented  with  the 
appropriate  rigor  and  detail  for  a  mature  service  organization. 

G.  The  Project  Monitoring  and  Control  process  area  adequately  addresses  service  level  management. 

H.  Material  about  the  collection  of  customer  satisfaction  information  is  adequately  covered  as  a  specific 
practice  in  Organizational  Service  Management  (an  optional  process  area)  and  as  informative  material 
in  the  Service  Delivery  process  area. 

I.  Maintenance  found  in  the  Service  Delivery  process  area  is  adequately  differentiated  from  product 
maintenance  covered  by  CMMI-DEV. 

j.  The  IPPD  addition  is  as  appropriate  or  as  applicable  for  CMMI-SVC  as  it  is  for  CMMI-DEV  and  should 
be  added. 

K.  The  Supplier  Agreement  Management  process  area  is  appropriate  both  for  organizations  with  tangible 
products  and  service  organizations  with  supplier  agreements  solely  for  services. 

L  The  Supplier  Agreement  Management  process  area  should  be  required  to  reach  maturity  level  2  for 
service  organizations  with  supplier  agreements  solely  for  services  (as  it  is  for  organizations  with 
suppliers  of  tangible  products). 
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•  CMMI-SVC  complements  ITIL 

•  Summarizes  ITIL  best  practices  into  a  small  set  of 
specific  practices. 

•  Reuses  about  80%  of  the  current  CMMI  model,  allowing 
users  to  leverage  their  investments  in  development- 
based  process  training,  improvements,  and  infrastructure 
to  serv  ce-based  offerings. 

•  Provides  an  industry-accepted  maturity  model,  helping 
organizations  to  plan  and  track  their  incremental 
progress  toward  high  maturity. 

•  Uses  the  same  SCAMPI  appraisal  method  that  is  used 
with  the  current  CMMI  mooel,  allowing  organizations  to 
leverage  appraisal  expertise,  preparation  methods,  and 
selected  artifacts. 
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Welcome  to  the  EN  Process  Improvement  Resource  Center 

Since  1998j  a  government-industry-Software  Engineering  Institute  (SEI)  cellaboration  has  been  under  way  to 
develop  a  product  suite  of  modelSj  training^  and  assessment  methodology  that  support  integrated  process 
and  product  improvement  across  the  enterprise,  These  products  are  intended  to  replace  legacy  maturity 
modeISjincluding  SW-CMM  and  Electronics  Industries  Association  Interim  Standard  (EIA/IS)  731,  the  Systems 
Engineering  Capability  Model  (SECM)  in  December  2003. 

Toolkits 

•  Configuration  Management  Toolkit  * 

•  Enterprise  Integration  Toolkit 

•  Integrated  Testing  Toolkit  * 

•  Life-Cycle  Logistics  Toolkit 

•  Partnering  Toolkit 

•  Quality  Assurance  Toolkit 

•  Reouirements  Process  Toolkit* 

•  Risk  Management  Toolkit  * 

•  System  Safety  Process  Toolkit 

•  Technical  Project  Planning  Toolkit 


□  NTACT  US 


*  CPSG  Focus  Areas 
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Step  1  -  CM  Process 


Action:  Appoint  an  Enterprise  Configuration  Manager  and  Component  System 
CMs  and  Develop  Implementation  Strategy 

A  Configuration  Manager  needs  to  be  appointed  for  the  program  as  well  as  a 
support  team  to  handle  the  Integrated  Digital  Environment  and  any  automated 
configuration  management  tools  to  be  used  on  the  program.  An 
implementation  strategy  needs  to  be  developed  that  addresses  the 
requirements  for  the  configuration  management  effort . 


Sub  Steps: 
Strategy 
Hierarchy 
Control 
Stakeholders 
Buy-in 


ESC  Actions 
(Optional) 
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Required 

The  major  steps  are  the  goals  of  each  process.  All  organizations  are  required 
to  implement  each  process  that  achieves  these  goals. 

Optional 

The  actions  (e.g.,  1a,  1b,  etc)  for  each  step  are  considered  best  practices  and 
are  expected  to  be  performed  by  each  organization  to  implement  satisfactory 
processes.  It  is  possible  to  satisfy  the  required  goals  without  implementing  the 
expected  practices  but  the  burden  of  proof  is  on  the  organization  using  an 
alternative  set  of  practices. 

Suggested 

All  material  covered  in  the  training  sessions  and  resources  provided  in  the 
toolkit  are  suggested  approaches  to  implementing  the  expected  practices.  This 
material  is  optional  and  may  be  used  at  the  discretion  of  the  organization. 
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3.2.  Configuration  Management  Practices 

Using  the  ESC  provided  guidance,  CPSG/EN  has  developed  a  Configuration  Management 
Implementation  Guide  which  provides  additional  guidance  when  developing  a  program  specific 
CM  process.  The  ten  areas  described  below  must  be  implemented. 

^^dl3.2.1.  Preparation  and  Hierarchy 

In  this  area,  each  acquisition  program,  system,  and  end  item  is  required  to: 

a.  Appoint,  in  witing,  a  configuration  manager 

b.  Develop  a  CM  plan 

c.  Identify  the  configuration  items  (CTs) 

d.  Identify’  the  internal  and  external  stakeholders 

e.  Determine  the  C'M  stnicture  and  hierarchy 

f  Establish  a  Configuration  Control  Board  (CCB) 

Infrastructure  and  Tools 

In  tins  area,  each  acquisition  programT^’stem.  and  end  item  is  required  to: 

a.  Identify’  tool  requirements 

b.  Coordinate  tool  requirements  with  CTS  GEN 

c.  Train  their  CM  workforce 

d.  Update  CM  information  on  ENWeb 
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A  Implementation  Guides 

T  Contain  the  fHowo 
T  Allowable  program  tailoring  identified 
I  Templates  provided  for  each  process  area 

V  Provide  Program  Managers/Lead  Engineers 
with  an  f80%6solution 

I  Ensure  consistency  across  CPSG 

V  Example:  Configuration  Management  Process 
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Plan  Development  and 
Tailoring  Guidance 


4.1.4  Detemiine  Configui  fition  Management  Stincture  and  Hierarchy 

A  govei'iuiient  configuration  inanageinent  progi’ain  needs  to  be  established  for  each  progi’am  to 
handle  tlie  user  requirements,  system  requirements,  and  extei'iial  interfaces.  For  each 
development  conti  act  au^arded  undei’  tlie  progi’am,  tlie  contr  actor  will  probably  be  required  to 
establish  a  configuration  management  program  to  handle  the  system  requirements,  allocated 
requirements,  product  requirements,  and  support  requirements.  Once  tire  system  is  fielded,  a 
sustainment  configuration  management  progr  am  needs  to  be  established.  In  a  system  of  systems 
progr  am  or  one  involving  multiple  development  conti’acts,  a  configuration  management  hiei’arcliy 
must  be  established. 

Implementation: 

CM  Plan  Template:  Configuration  Management  Organization 

Tailoiing  Guidance: 

Hiis  is  always  required,  but  tlie  actual  structure  is  left  to  tlie  progi'ani. 


UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #20 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Agenda 


ACryptologic  Systems  Group  (CPSG) 
AElectronic  Systems  Center  CMMI  Focus 
ACPSG  CMMI  Implementation 
I  Process  Guide 
T  Implementation  Guides 

ACPSG  CMMI  Compliance 
ACPSG  Training 


UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #21 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


ESC/EN  Web 


(/) 

OS 

0) 

< 

10 

to 

(1) 

o 

o 


O 

O 

C/D 

LU 


CPSG  CMMI  Responses 


Requirements  Development  and  Management 

Configuration  Management 

Risk  Management 

Enterprise  Integration 

Integrated  Testing 

Technical  Project  Planning 

Quality  Assurance  Process 

System  Safety  Process 

Life-Cycle  Logistics 


Level  2:  Complete 
Level  2:  Complete 
Level  2:  Complete 
Level  2:  11  Mar.  07 
Level  2:  04  Feb.  07 
Level  2:  Complete 
Level  2:  04  Feb.  07 
Level  2:  07  Jan.  07 
Level  2:  Complete 


Status:  Q 
Status:  (b) 
Status:  @ 
Status:  (J 
Status:  (e) 
Status:  (b) 
Status:  @ 
Status:  (J) 
Status: 


Updated: 

Updated: 

Updated: 


Updated: 


Updated: 

Updated: 

Updated: 

Updated: 


Updated: 


UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #22 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


upgrade  to 

s  and  Expanded  Features 


sc  CMMI  (ENWeb) 
Generic  Goals 


tn 

OS 

o 

O 

o 

o 

o 

0) 

(f) 

c 

LJU 

(1) 

O 

Part  M:  Geiieiic  Practices 

Achieve  Specific  Goals 
^1.1:  HertormlJevelopment  Base  Practices 

GP  1.2:  Perform  Management  Base  Practices 

:  Institiitionalize  a  Manarjed  Process 

GG  2.1 :  For  Requirements  Development 


Lev:  2 
Lev:  2 


Yes 

Yes 


GP  2.1.1:  Establish  an  Organizational  Policy 

Lev:  2 

O 

No 

2B  Feb.  07 

GP  2.1.2:  Plan  the  Process 

Lev:  2 

Yes 

GP  2.1.3:  Provide  Resources 

Lev:  2 

Yes 

GP  2.1.4:  Assign  Responsibility 

Lev:  2 

Yes 

GP  2.1.5:  Train  People 

Lev:  2 

Yes 

GP  2.1.EI:  Manage  Configurations 

Lev:  2 

O 

No 

2B  Feb.  07 

GP  2.1.7:  Identify  and  Involve  Relevant  Stakeholders 

Lev:  2 

Partial 

31  Jan.  07 

GP  2.1.B:  Monitor  and  Control  the  Process 

Lev:  2 

Yes 

GP  2.1.9:  Objectively  Evaluate  Adherence 

Lev:  2 

o 

No 

01  Jun.  07 

GP  2.1 .10:  Review  Status  with  Higher-Level  Management 

Lev:  2 

o 

No 

01  Mar.  07 

GP  2.1 .1 1 :  Perform  Base  Practices 

Lev:  2 

o 

No 

01  Jun.  07 

UNCLASSIFIED 


“Securing  the  Global  Information  Grid  (GIG)” 


Slide  #23 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


upgrade  to 
s  and  Expanded  Features 


ESC  CMMI  (ENWeb) 
nequiiements  Process  Specific  Goals 


Requirements  Development  and  Management 

Level  2:  01  Jun.  07 

Status: 

Question 

Answer  Planned  Date  Status 

tn 

frt 

Part  1:  Specific  Practices 

VVJ 

o 

Cl^1:  Deveiop  Customer  Requirement^ 

o 

SP  1.1:  Collect  Stakeholder  Needs 

Lev:  2 

Yes 

© 

Edit 

o 

SP  1.2:  Transform  Stakeholder  Needs,  Expectations, 

Lev:  2 

Yes 

0 

Edit 

G) 

Q. 

Constraints,  and  Interfaces  into  Customer  Requirements 

CO 

SP  1 .3:  Elicit  Needs 

Lev:  2 

Yes 

0 

Edit 

1 

CS^2:  Develop  Product  Requirement^ 

o 

SF^2.1:  Establish  Product  and  Product  Component 

Lev:  2 

Yes 

© 

Edit 

o 

CO 

Requirements 

LU 

SP  2.2:  Allocate  Product  Component  Requirements 

Lev:  2 

Yes 

© 

Edit 

SP  2.3:  Identify  Interface  Requirements 

Lev:  2 

Yes 

© 

Edit 

UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #24 


Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


ESC/EN  Web 


C/) 

CO 

CD 

< 

(/) 

C/) 

CD 

O 

O 

CL 

B 

CO 

o 

Q. 

O 

O 


CPSG  CMMI  Responses 


Requirements  Development  and  Management 

Level  2:  Complete 

Status: 

© 

Updated: 

Configuration  Management 

Level  2:  Complete 

Status: 

© 

Updated:  BfflBiliBI 

Risk  Management 

Level  2:  Complete 

Status: 

© 

Updated:  SWiia 

Cjn^i'Pi'ise  Integration^ 

Level  2:  11  Mar.  07 

Status: 

© 

Updated: 

Integrated  Testing 

Level  2:  04  Feb.  07 

Status: 

© 

Updated:  IMUPH 

Technical  Project  Planning 

Level  2:  Complete 

Status: 

© 

Updated: 

CQuali^  Assurance  Pro£e2^ 

Level  2:  04  Feb.  07 

Status: 

© 

Updated: 

CSystem  Safety  Process^ 

Level  2:  07  Jan.  07 

Status: 

© 

Updated:  IBM 

Life-Cycle  Logistics 

Level  2:  Complete 

Status: 

© 

Updated: 

UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #25 


^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Agenda 


ACryptologic  Systems  Group  (CPSG) 
AElectronic  Systems  Center  Focus 
ACPSG  CMMI  Implementation 
I  Process  Guide 
T  Implementation  Guides 

ACPSG  CMMI  Compliance 
ACPSG  Training 


UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #26 


^PDF 

^  Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Training  Plan 


UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)”  Slide  #27 


^PDF 

Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


Wrap  -  Up 


ACryptologic  Systems 
AElectronic  System^ 
ACPSG  CMMI  Im 


PSG) 

CMMI  Focus 
tion 


T  Process  Quid 
V  Impleme 

ACPS 
ACP 


S 

pliance 


UNCLASSIFIED 


“Securing  the  Global  Information  Grid  (GIG)” 


Slide  #28 


-  PDF 

Complete 


Your  complimentary 
use  period  has  ended. 
Thank  you  for  using 
PDF  Complete. 


fade  to 

and  Expanded  Features 


hank  You  for  Your  Time 


UNCLASSIFIED  “Securing  the  Global  Information  Grid  (GIG)” 


Slide  #29 


